2024-09-15T00:24:33.349 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T00:24:33.392 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T00:24:33.478 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845 branch: squid description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: ceph-qa@ceph.com first_in_suite: false job_id: '7904845' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 4271 sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 sleep_before_teardown: 0 subset: 97/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJKSh+HMyOt74GwrbujlVZhlgghNpatFlppFK6XAf1VJihRqZxdonpdkCxEi7ha9wNl2CQSxJFWt7DBlaGLl+4M= smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLJXXwvZdSj6pyFoClf2ZtIjKiyS/8t92Tv/1J/74+BOEC17P72ZXMB9q9sOaQ2LdvXVT8SeKmJQ75utNULR7Ok= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-14_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T00:24:33.478 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa; will attempt to use it 2024-09-15T00:24:33.479 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa/tasks 2024-09-15T00:24:33.479 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T00:24:33.480 INFO:teuthology.task.internal:Checking packages... 2024-09-15T00:24:33.500 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ecab7be58963a5bd03910cd73eef66405d7fdd50' 2024-09-15T00:24:33.500 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T00:24:33.500 INFO:teuthology.packaging:ref: None 2024-09-15T00:24:33.500 INFO:teuthology.packaging:tag: None 2024-09-15T00:24:33.500 INFO:teuthology.packaging:branch: squid 2024-09-15T00:24:33.500 INFO:teuthology.packaging:sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-15T00:24:33.501 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-15T00:24:33.732 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-324.gecab7be5 2024-09-15T00:24:33.734 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T00:24:33.739 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T00:24:33.739 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T00:24:33.769 INFO:teuthology.task.internal:Saving configuration 2024-09-15T00:24:33.786 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T00:24:33.803 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T00:24:33.826 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 00:19:56.545889', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJKSh+HMyOt74GwrbujlVZhlgghNpatFlppFK6XAf1VJihRqZxdonpdkCxEi7ha9wNl2CQSxJFWt7DBlaGLl+4M='} 2024-09-15T00:24:33.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 00:19:56.544487', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLJXXwvZdSj6pyFoClf2ZtIjKiyS/8t92Tv/1J/74+BOEC17P72ZXMB9q9sOaQ2LdvXVT8SeKmJQ75utNULR7Ok='} 2024-09-15T00:24:33.849 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T00:24:33.853 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-15T00:24:33.853 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-15T00:24:33.853 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T00:24:33.942 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6b896c7ac0>, signals=[15]) 2024-09-15T00:24:33.943 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T00:24:33.947 INFO:teuthology.task.internal:Opening connections... 2024-09-15T00:24:33.947 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2024-09-15T00:24:33.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T00:24:34.025 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T00:24:34.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T00:24:34.102 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T00:24:34.108 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2024-09-15T00:24:34.126 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2024-09-15T00:24:34.126 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2024-09-15T00:24:34.185 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Stream" 2024-09-15T00:24:34.185 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="9" 2024-09-15T00:24:34.185 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2024-09-15T00:24:34.185 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2024-09-15T00:24:34.185 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="9" 2024-09-15T00:24:34.185 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el9" 2024-09-15T00:24:34.186 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T00:24:34.186 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2024-09-15T00:24:34.186 INFO:teuthology.orchestra.run.smithi002.stdout:LOGO="fedora-logo-icon" 2024-09-15T00:24:34.186 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T00:24:34.186 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://centos.org/" 2024-09-15T00:24:34.186 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T00:24:34.186 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T00:24:34.186 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T00:24:34.187 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2024-09-15T00:24:34.209 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-09-15T00:24:34.228 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-09-15T00:24:34.229 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-09-15T00:24:34.287 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2024-09-15T00:24:34.287 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="9" 2024-09-15T00:24:34.287 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2024-09-15T00:24:34.287 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2024-09-15T00:24:34.287 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="9" 2024-09-15T00:24:34.287 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el9" 2024-09-15T00:24:34.288 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T00:24:34.288 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2024-09-15T00:24:34.288 INFO:teuthology.orchestra.run.smithi059.stdout:LOGO="fedora-logo-icon" 2024-09-15T00:24:34.288 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T00:24:34.288 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2024-09-15T00:24:34.288 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T00:24:34.289 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T00:24:34.289 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T00:24:34.289 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-09-15T00:24:34.309 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T00:24:34.317 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T00:24:34.345 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T00:24:34.346 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2024-09-15T00:24:34.348 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-09-15T00:24:34.365 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T00:24:34.379 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T00:24:34.379 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2024-09-15T00:24:34.406 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-09-15T00:24:34.430 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T00:24:34.491 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T00:24:34.505 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T00:24:34.505 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T00:24:34.506 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T00:24:34.506 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T00:24:34.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T00:24:34.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T00:24:34.524 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-09-15T00:24:34.524 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-09-15T00:24:34.580 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-503.el9.x86_64 2024-09-15T00:24:34.581 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.14.0-503.el9.x86_64 2024-09-15T00:24:34.581 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2024-09-15T00:24:34.582 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-503.el9.x86_64 2024-09-15T00:24:34.582 INFO:teuthology.task.kernel:Running kernel on smithi002: 5.14.0-503.el9.x86_64 2024-09-15T00:24:34.582 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum install -y kernel 2024-09-15T00:24:36.883 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-15T00:24:38.084 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-09-15T00:24:38.732 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-15T00:24:40.002 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-15T00:24:43.159 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-15T00:24:44.660 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 19 kB 00:00 2024-09-15T00:24:45.564 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-15T00:24:46.777 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.5 MB 00:02 2024-09-15T00:24:48.328 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 19 kB 00:00 2024-09-15T00:24:49.433 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-15T00:24:51.120 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-15T00:24:53.219 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T00:24:53.219 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T00:24:53.301 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:24:53.311 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:24:53.312 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-09-15T00:24:53.312 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:24:53.312 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:24:53.312 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T00:24:53.312 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-15T00:24:53.312 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T00:24:53.312 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T00:24:53.312 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T00:24:53.312 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:24:53.312 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:24:53.313 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:24:53.313 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2024-09-15T00:24:53.313 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:24:53.313 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 79 M 2024-09-15T00:24:53.313 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 126 M 2024-09-15T00:24:53.313 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:24:53.497 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 267 kB/s | 23 kB 00:00 2024-09-15T00:24:53.847 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-09-15T00:24:54.130 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 45 MB/s | 29 MB 00:00 2024-09-15T00:24:54.481 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 32 MB/s | 35 MB 00:01 2024-09-15T00:24:54.481 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:24:54.481 INFO:teuthology.orchestra.run.smithi059.stdout:Total 67 MB/s | 79 MB 00:01 2024-09-15T00:24:55.030 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-15T00:24:55.188 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:24:55.577 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:24:55.577 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:24:56.558 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:24:56.558 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:24:57.153 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T00:24:57.153 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T00:24:57.243 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:24:57.250 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:24:57.250 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout:Install 4 Packages 2024-09-15T00:24:57.251 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:24:57.252 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 79 M 2024-09-15T00:24:57.252 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 126 M 2024-09-15T00:24:57.252 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:24:57.814 INFO:teuthology.orchestra.run.smithi002.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 42 kB/s | 23 kB 00:00 2024-09-15T00:24:59.132 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:24:59.807 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T00:24:59.889 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T00:25:00.198 INFO:teuthology.orchestra.run.smithi002.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.4 MB/s | 16 MB 00:02 2024-09-15T00:25:00.948 INFO:teuthology.orchestra.run.smithi002.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 9.1 MB/s | 29 MB 00:03 2024-09-15T00:25:01.689 INFO:teuthology.orchestra.run.smithi002.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 7.8 MB/s | 35 MB 00:04 2024-09-15T00:25:01.690 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:25:01.690 INFO:teuthology.orchestra.run.smithi002.stdout:Total 18 MB/s | 79 MB 00:04 2024-09-15T00:25:02.402 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:25:02.688 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T00:25:02.782 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:25:02.782 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:25:02.826 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T00:25:03.768 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:25:03.768 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:25:06.317 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:25:06.974 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T00:25:07.056 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T00:25:07.956 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T00:25:08.001 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:09.863 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T00:25:09.993 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T00:25:13.145 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:15.114 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T00:25:15.160 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:20.308 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:52.507 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:52.549 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T00:25:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T00:25:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T00:25:53.760 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:53.760 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:25:53.760 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:25:53.760 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T00:25:53.760 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T00:25:53.761 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T00:25:53.761 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T00:25:53.761 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:25:53.761 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:25:54.072 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T00:25:54.163 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T00:25:54.164 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T00:25:54.164 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T00:25:54.164 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T00:25:54.164 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T00:25:54.164 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-15T00:25:54.165 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T00:25:54.165 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T00:25:54.223 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T00:25:54.223 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T00:25:54.223 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2024-09-15T00:25:54.254 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2024-09-15T00:25:54.254 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2024-09-15T00:25:54.254 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.06 2024-09-15T00:25:54.254 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 82.el9 2024-09-15T00:25:54.254 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2024-09-15T00:25:54.254 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T00:25:54.254 INFO:teuthology.orchestra.run.smithi059.stdout:Group : Unspecified 2024-09-15T00:25:54.254 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 8274670 2024-09-15T00:25:54.255 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2024-09-15T00:25:54.255 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T00:25:54.255 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T00:25:54.255 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T00:25:54.255 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T00:25:54.255 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : builder@centos.org 2024-09-15T00:25:54.255 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2024-09-15T00:25:54.255 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T00:25:54.255 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2024-09-15T00:25:54.255 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2024-09-15T00:25:54.256 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:25:54.256 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T00:25:54.256 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T00:25:54.256 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T00:25:54.256 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2024-09-15T00:25:54.256 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:25:54.256 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T00:25:54.257 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T00:25:54.257 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 5.14.0-505.el9.x86_64 2024-09-15T00:25:54.258 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T00:25:54.918 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-09-15T00:25:56.247 INFO:teuthology.orchestra.run.smithi059.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T00:25:56.291 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-09-15T00:25:56.293 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-09-15T00:25:56.308 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.Il4mZi2K23 2024-09-15T00:25:56.308 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Il4mZi2K23 2024-09-15T00:25:56.384 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.Il4mZi2K23 2024-09-15T00:25:56.521 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.Il4mZi2K23 is 6KB 2024-09-15T00:25:56.570 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.Il4mZi2K23 2024-09-15T00:25:56.585 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T00:25:56.647 INFO:teuthology.orchestra.run.smithi059.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T00:25:56.647 INFO:teuthology.orchestra.run.smithi059.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T00:25:56.647 INFO:teuthology.orchestra.run.smithi059.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T00:25:56.647 INFO:teuthology.orchestra.run.smithi059.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T00:25:56.648 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T00:25:56.802 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-09-15T00:25:59.287 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:59.329 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:59.629 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:25:59.630 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T00:25:59.630 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T00:25:59.630 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T00:26:00.501 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T00:26:00.501 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:26:00.501 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:26:00.502 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T00:26:00.502 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T00:26:00.502 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T00:26:00.502 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T00:26:00.502 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:26:00.502 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:26:00.825 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T00:26:00.927 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T00:26:00.928 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T00:26:00.928 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T00:26:00.928 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T00:26:00.928 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T00:26:00.928 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-15T00:26:00.928 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T00:26:00.928 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T00:26:00.987 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T00:26:00.988 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T00:26:00.988 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2024-09-15T00:26:01.019 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2024-09-15T00:26:01.019 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2024-09-15T00:26:01.019 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.06 2024-09-15T00:26:01.019 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 82.el9 2024-09-15T00:26:01.019 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2024-09-15T00:26:01.019 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T00:26:01.019 INFO:teuthology.orchestra.run.smithi002.stdout:Group : Unspecified 2024-09-15T00:26:01.019 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 8274670 2024-09-15T00:26:01.019 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : builder@centos.org 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : CentOS 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T00:26:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T00:26:01.021 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2024-09-15T00:26:01.021 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:26:01.021 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T00:26:01.021 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T00:26:01.022 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 5.14.0-505.el9.x86_64 2024-09-15T00:26:01.022 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T00:26:01.691 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2024-09-15T00:26:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T00:26:03.099 INFO:teuthology.orchestra.run.smithi002.stderr:done 2024-09-15T00:26:03.102 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-09-15T00:26:03.116 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.MKRMBZ8tbB 2024-09-15T00:26:03.117 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MKRMBZ8tbB 2024-09-15T00:26:03.183 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.MKRMBZ8tbB 2024-09-15T00:26:03.324 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.MKRMBZ8tbB is 6KB 2024-09-15T00:26:03.372 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.MKRMBZ8tbB 2024-09-15T00:26:03.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T00:26:03.452 INFO:teuthology.orchestra.run.smithi002.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T00:26:03.452 INFO:teuthology.orchestra.run.smithi002.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T00:26:03.452 INFO:teuthology.orchestra.run.smithi002.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T00:26:03.452 INFO:teuthology.orchestra.run.smithi002.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T00:26:03.453 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T00:26:03.609 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2024-09-15T00:26:26.817 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-15T00:26:26.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-15T00:26:26.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T00:26:33.619 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-15T00:26:33.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-15T00:26:33.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T00:27:26.877 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T00:27:33.626 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T00:27:35.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-15T00:27:35.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T00:27:36.335 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-09-15T00:27:36.870 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-15T00:27:36.871 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T00:27:36.871 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-09-15T00:27:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-505.el9.x86_64 2024-09-15T00:27:36.888 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T00:27:36.888 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T00:27:36.888 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T00:27:37.889 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-15T00:27:37.889 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T00:27:37.965 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-09-15T00:27:37.992 DEBUG:teuthology.parallel:result is None 2024-09-15T00:27:42.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-15T00:27:42.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T00:27:43.118 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-09-15T00:27:43.471 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-15T00:27:43.471 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T00:27:43.471 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-09-15T00:27:43.486 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-505.el9.x86_64 2024-09-15T00:27:43.486 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T00:27:43.486 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T00:27:43.486 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T00:27:44.487 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-15T00:27:44.487 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T00:27:44.575 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2024-09-15T00:27:44.596 DEBUG:teuthology.parallel:result is None 2024-09-15T00:27:44.596 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T00:27:44.604 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T00:27:44.604 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T00:27:44.607 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T00:27:44.626 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T00:27:44.632 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T00:27:44.682 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T00:27:44.682 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T00:27:44.684 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T00:27:44.715 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T00:27:44.731 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T00:27:44.731 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T00:27:44.742 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T00:27:44.765 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T00:27:44.776 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T00:27:44.788 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T00:27:44.799 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T00:27:44.800 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T00:27:44.808 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T00:27:44.808 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T00:27:44.817 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T00:27:44.870 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T00:27:44.880 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T00:27:44.881 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T00:27:44.883 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T00:27:44.930 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T00:27:44.978 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T00:27:45.026 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T00:27:45.103 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T00:27:45.144 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:27:45.144 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T00:27:45.207 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T00:27:45.250 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T00:27:45.300 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T00:27:45.386 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T00:27:45.434 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:27:45.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T00:27:45.502 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2024-09-15T00:27:45.505 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-09-15T00:27:45.562 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T00:27:45.601 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T00:27:46.027 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T00:27:46.035 INFO:teuthology.task.internal:Starting timer... 2024-09-15T00:27:46.035 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T00:27:46.101 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T00:27:46.143 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-15T00:27:46.144 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2024-09-15T00:27:46.268 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2024-09-15T00:27:46.270 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2024-09-15T00:27:46.336 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2024-09-15T00:27:46.338 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T00:27:46.339 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2024-09-15T00:27:46.401 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2024-09-15T00:27:46.402 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2024-09-15T00:27:46.435 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2024-09-15T00:27:46.436 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi059.front.sepia.ceph.com': 'permissive'} 2024-09-15T00:27:46.436 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T00:27:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T00:27:46.502 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T00:27:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T00:27:46.529 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T00:27:46.529 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2024-09-15T00:27:46.598 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2024-09-15T00:27:46.637 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T00:27:46.648 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T00:27:46.661 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T00:27:46.661 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi059.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T00:31:38.793 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi059.front.sepia.ceph.com')] 2024-09-15T00:31:38.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-15T00:31:38.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T00:31:38.874 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-09-15T00:31:38.954 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-15T00:31:38.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-15T00:31:38.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T00:31:39.035 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-09-15T00:31:39.118 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-15T00:31:39.118 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T00:31:39.129 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T00:31:39.129 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T00:31:39.129 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T00:31:39.132 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T00:31:39.132 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T00:31:39.164 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T00:31:39.181 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T00:31:39.190 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T00:31:39.206 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T00:31:39.207 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2024-09-15T00:31:39.219 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2024-09-15T00:31:39.233 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2024-09-15T00:31:39.235 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T00:31:39.245 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2024-09-15T00:31:39.250 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T00:31:39.260 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T00:31:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T00:31:39.310 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-09-15T00:31:39.313 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T00:31:39.314 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-09-15T00:31:39.314 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T00:31:39.314 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T00:31:39.314 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T00:31:39.314 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T00:31:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-09-15T00:31:39.347 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T00:31:39.347 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-09-15T00:31:39.347 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T00:31:39.347 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T00:31:39.347 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T00:31:39.347 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T00:31:39.348 INFO:teuthology.run_tasks:Running task install... 2024-09-15T00:31:39.356 DEBUG:teuthology.task.install:project ceph 2024-09-15T00:31:39.356 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}} 2024-09-15T00:31:39.357 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'} 2024-09-15T00:31:39.357 INFO:teuthology.task.install:Using flavor: default 2024-09-15T00:31:39.362 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T00:31:39.363 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-15T00:31:39.363 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'tag': None, 'wait_for_package': False} 2024-09-15T00:31:39.363 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-15T00:31:39.365 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'tag': None, 'wait_for_package': False} 2024-09-15T00:31:39.366 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-15T00:31:39.509 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/ 2024-09-15T00:31:39.509 INFO:teuthology.task.install.rpm:Package version is 19.1.1-324.gecab7be5 2024-09-15T00:31:39.513 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/ 2024-09-15T00:31:39.513 INFO:teuthology.task.install.rpm:Package version is 19.1.1-324.gecab7be5 2024-09-15T00:31:39.621 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T00:31:39.622 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:31:39.622 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T00:31:39.630 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T00:31:39.631 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:31:39.631 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T00:31:39.650 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-15T00:31:39.650 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ecab7be58963a5bd03910cd73eef66405d7fdd50/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T00:31:39.657 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-15T00:31:39.658 DEBUG:teuthology.orchestra.run.smithi002:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ecab7be58963a5bd03910cd73eef66405d7fdd50/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T00:31:39.717 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T00:31:39.725 DEBUG:teuthology.orchestra.run.smithi002:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T00:31:39.795 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T00:31:39.802 DEBUG:teuthology.orchestra.run.smithi002:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T00:31:39.860 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2024-09-15T00:31:39.862 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2024-09-15T00:31:39.867 INFO:teuthology.orchestra.run.smithi002.stdout:check_obsoletes = 1 2024-09-15T00:31:39.869 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum clean all 2024-09-15T00:31:40.107 INFO:teuthology.orchestra.run.smithi059.stdout:45 files removed 2024-09-15T00:31:40.114 INFO:teuthology.orchestra.run.smithi002.stdout:45 files removed 2024-09-15T00:31:40.134 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2024-09-15T00:31:40.141 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-radosgw 2024-09-15T00:31:40.777 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 243 kB/s | 84 kB 00:00 2024-09-15T00:31:40.781 INFO:teuthology.orchestra.run.smithi002.stdout:ceph packages for x86_64 244 kB/s | 84 kB 00:00 2024-09-15T00:31:41.061 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-15T00:31:41.098 INFO:teuthology.orchestra.run.smithi002.stdout:ceph noarch packages 43 kB/s | 12 kB 00:00 2024-09-15T00:31:41.336 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-15T00:31:41.391 INFO:teuthology.orchestra.run.smithi002.stdout:ceph source packages 7.4 kB/s | 1.9 kB 00:00 2024-09-15T00:31:41.820 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-15T00:31:43.116 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.2 MB 00:01 2024-09-15T00:31:43.829 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-15T00:31:45.042 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-15T00:31:49.097 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.5 MB 00:01 2024-09-15T00:31:50.589 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 86 kB/s | 19 kB 00:00 2024-09-15T00:31:51.310 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 38 MB/s | 23 MB 00:00 2024-09-15T00:31:51.642 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.5 MB 00:02 2024-09-15T00:31:53.201 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 19 kB 00:00 2024-09-15T00:31:54.089 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-15T00:31:56.982 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-15T00:31:59.411 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:31:59.423 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:31:59.423 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-15T00:31:59.423 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:31:59.423 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:31:59.423 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:19.1.1-324.gecab7be5.el9 ceph 11 M 2024-09-15T00:31:59.423 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2024-09-15T00:31:59.424 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.4 M 2024-09-15T00:31:59.424 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.2 M 2024-09-15T00:31:59.424 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-15T00:31:59.424 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T00:31:59.424 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:19.1.1-324.gecab7be5.el9 ceph 5.4 M 2024-09-15T00:31:59.424 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:19.1.1-324.gecab7be5.el9 ceph 21 M 2024-09-15T00:31:59.424 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:19.1.1-324.gecab7be5.el9 ceph 25 k 2024-09-15T00:31:59.424 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T00:31:59.424 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T00:31:59.424 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 752 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 507 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 5.3 M 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 45 k 2024-09-15T00:31:59.425 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:19.1.1-324.gecab7be5.el9 ceph 136 k 2024-09-15T00:31:59.426 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:19.1.1-324.gecab7be5.el9 ceph 162 k 2024-09-15T00:31:59.426 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T00:31:59.426 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:19.1.1-324.gecab7be5.el9 ceph 321 k 2024-09-15T00:31:59.426 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 301 k 2024-09-15T00:31:59.426 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:19.1.1-324.gecab7be5.el9 ceph 99 k 2024-09-15T00:31:59.426 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T00:31:59.426 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T00:31:59.426 INFO:teuthology.orchestra.run.smithi059.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T00:31:59.426 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T00:31:59.426 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2024-09-15T00:31:59.427 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T00:31:59.427 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:31:59.427 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:31:59.427 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:31:59.427 INFO:teuthology.orchestra.run.smithi059.stdout:Install 30 Packages 2024-09-15T00:31:59.427 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2024-09-15T00:31:59.427 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:31:59.428 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 60 M 2024-09-15T00:31:59.428 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:31:59.711 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-15T00:32:00.442 INFO:teuthology.orchestra.run.smithi059.stdout:(1/32): ceph-base-19.1.1-324.gecab7be5.el9.x86_ 10 MB/s | 5.4 MB 00:00 2024-09-15T00:32:00.509 INFO:teuthology.orchestra.run.smithi059.stdout:(2/32): ceph-selinux-19.1.1-324.gecab7be5.el9.x 378 kB/s | 25 kB 00:00 2024-09-15T00:32:00.593 INFO:teuthology.orchestra.run.smithi059.stdout:(3/32): libcephfs2-19.1.1-324.gecab7be5.el9.x86 8.8 MB/s | 752 kB 00:00 2024-09-15T00:32:00.676 INFO:teuthology.orchestra.run.smithi059.stdout:(4/32): libradosstriper1-19.1.1-324.gecab7be5.e 6.0 MB/s | 507 kB 00:00 2024-09-15T00:32:00.801 INFO:teuthology.orchestra.run.smithi059.stdout:(5/32): ceph-radosgw-19.1.1-324.gecab7be5.el9.x 12 MB/s | 11 MB 00:00 2024-09-15T00:32:00.859 INFO:teuthology.orchestra.run.smithi059.stdout:(6/32): python3-ceph-argparse-19.1.1-324.gecab7 778 kB/s | 45 kB 00:00 2024-09-15T00:32:00.935 INFO:teuthology.orchestra.run.smithi059.stdout:(7/32): python3-ceph-common-19.1.1-324.gecab7be 1.8 MB/s | 136 kB 00:00 2024-09-15T00:32:01.043 INFO:teuthology.orchestra.run.smithi059.stdout:(8/32): librgw2-19.1.1-324.gecab7be5.el9.x86_64 15 MB/s | 5.3 MB 00:00 2024-09-15T00:32:01.069 INFO:teuthology.orchestra.run.smithi059.stdout:(9/32): python3-cephfs-19.1.1-324.gecab7be5.el9 1.2 MB/s | 162 kB 00:00 2024-09-15T00:32:01.102 INFO:teuthology.orchestra.run.smithi059.stdout:(10/32): python3-rados-19.1.1-324.gecab7be5.el9 5.4 MB/s | 321 kB 00:00 2024-09-15T00:32:01.127 INFO:teuthology.orchestra.run.smithi059.stdout:(11/32): python3-rbd-19.1.1-324.gecab7be5.el9.x 5.0 MB/s | 301 kB 00:00 2024-09-15T00:32:01.153 INFO:teuthology.orchestra.run.smithi059.stdout:(12/32): python3-rgw-19.1.1-324.gecab7be5.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-15T00:32:01.345 INFO:teuthology.orchestra.run.smithi059.stdout:(13/32): ceph-common-19.1.1-324.gecab7be5.el9.x 15 MB/s | 21 MB 00:01 2024-09-15T00:32:01.912 INFO:teuthology.orchestra.run.smithi059.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 80 kB/s | 45 kB 00:00 2024-09-15T00:32:01.962 INFO:teuthology.orchestra.run.smithi059.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 128 kB/s | 107 kB 00:00 2024-09-15T00:32:02.021 INFO:teuthology.orchestra.run.smithi059.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 185 kB/s | 160 kB 00:00 2024-09-15T00:32:02.199 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:32:02.210 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-09-15T00:32:02.210 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:02.210 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw x86_64 2:19.1.1-324.gecab7be5.el9 ceph 11 M 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.4 M 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.2 M 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base x86_64 2:19.1.1-324.gecab7be5.el9 ceph 5.4 M 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common x86_64 2:19.1.1-324.gecab7be5.el9 ceph 21 M 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux x86_64 2:19.1.1-324.gecab7be5.el9 ceph 25 k 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T00:32:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 752 k 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 507 k 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 5.3 M 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T00:32:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 45 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common x86_64 2:19.1.1-324.gecab7be5.el9 ceph 136 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs x86_64 2:19.1.1-324.gecab7be5.el9 ceph 162 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados x86_64 2:19.1.1-324.gecab7be5.el9 ceph 321 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 301 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw x86_64 2:19.1.1-324.gecab7be5.el9 ceph 99 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout:Installing weak dependencies: 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T00:32:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:02.214 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:32:02.214 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:02.214 INFO:teuthology.orchestra.run.smithi002.stdout:Install 30 Packages 2024-09-15T00:32:02.214 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 2 Packages 2024-09-15T00:32:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:02.214 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 60 M 2024-09-15T00:32:02.214 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:32:02.587 INFO:teuthology.orchestra.run.smithi059.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 75 kB/s | 42 kB 00:00 2024-09-15T00:32:02.749 INFO:teuthology.orchestra.run.smithi059.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 373 kB/s | 292 kB 00:00 2024-09-15T00:32:02.780 INFO:teuthology.orchestra.run.smithi059.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 764 kB/s | 662 kB 00:00 2024-09-15T00:32:03.163 INFO:teuthology.orchestra.run.smithi002.stdout:(1/32): ceph-base-19.1.1-324.gecab7be5.el9.x86_ 8.0 MB/s | 5.4 MB 00:00 2024-09-15T00:32:03.204 INFO:teuthology.orchestra.run.smithi002.stdout:(2/32): ceph-selinux-19.1.1-324.gecab7be5.el9.x 612 kB/s | 25 kB 00:00 2024-09-15T00:32:03.296 INFO:teuthology.orchestra.run.smithi059.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 65 kB/s | 36 kB 00:00 2024-09-15T00:32:03.313 INFO:teuthology.orchestra.run.smithi002.stdout:(3/32): libcephfs2-19.1.1-324.gecab7be5.el9.x86 6.8 MB/s | 752 kB 00:00 2024-09-15T00:32:03.347 INFO:teuthology.orchestra.run.smithi059.stdout:(21/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.1 MB/s | 308 kB 00:00 2024-09-15T00:32:03.372 INFO:teuthology.orchestra.run.smithi059.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 80 kB/s | 47 kB 00:00 2024-09-15T00:32:03.414 INFO:teuthology.orchestra.run.smithi059.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-09-15T00:32:03.439 INFO:teuthology.orchestra.run.smithi059.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-15T00:32:03.446 INFO:teuthology.orchestra.run.smithi002.stdout:(4/32): ceph-radosgw-19.1.1-324.gecab7be5.el9.x 11 MB/s | 11 MB 00:00 2024-09-15T00:32:03.464 INFO:teuthology.orchestra.run.smithi059.stdout:(25/32): qatlib-24.02.0-1.el9.x86_64.rpm 252 kB/s | 221 kB 00:00 2024-09-15T00:32:03.472 INFO:teuthology.orchestra.run.smithi002.stdout:(5/32): libradosstriper1-19.1.1-324.gecab7be5.e 3.1 MB/s | 507 kB 00:00 2024-09-15T00:32:03.490 INFO:teuthology.orchestra.run.smithi059.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-15T00:32:03.514 INFO:teuthology.orchestra.run.smithi002.stdout:(6/32): python3-ceph-argparse-19.1.1-324.gecab7 1.1 MB/s | 45 kB 00:00 2024-09-15T00:32:03.523 INFO:teuthology.orchestra.run.smithi059.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 7.4 MB/s | 191 kB 00:00 2024-09-15T00:32:03.598 INFO:teuthology.orchestra.run.smithi059.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 22 MB/s | 1.6 MB 00:00 2024-09-15T00:32:03.640 INFO:teuthology.orchestra.run.smithi059.stdout:(29/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.7 MB/s | 839 kB 00:00 2024-09-15T00:32:03.716 INFO:teuthology.orchestra.run.smithi059.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-15T00:32:03.730 INFO:teuthology.orchestra.run.smithi002.stdout:(7/32): ceph-common-19.1.1-324.gecab7be5.el9.x8 17 MB/s | 21 MB 00:01 2024-09-15T00:32:03.756 INFO:teuthology.orchestra.run.smithi002.stdout:(8/32): python3-ceph-common-19.1.1-324.gecab7be 563 kB/s | 136 kB 00:00 2024-09-15T00:32:03.790 INFO:teuthology.orchestra.run.smithi002.stdout:(9/32): python3-cephfs-19.1.1-324.gecab7be5.el9 2.7 MB/s | 162 kB 00:00 2024-09-15T00:32:03.815 INFO:teuthology.orchestra.run.smithi002.stdout:(10/32): python3-rados-19.1.1-324.gecab7be5.el9 5.4 MB/s | 321 kB 00:00 2024-09-15T00:32:03.908 INFO:teuthology.orchestra.run.smithi059.stdout:(31/32): librbd1-19.1.1-324.gecab7be5.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-15T00:32:03.917 INFO:teuthology.orchestra.run.smithi002.stdout:(11/32): librgw2-19.1.1-324.gecab7be5.el9.x86_6 11 MB/s | 5.3 MB 00:00 2024-09-15T00:32:03.943 INFO:teuthology.orchestra.run.smithi002.stdout:(12/32): python3-rgw-19.1.1-324.gecab7be5.el9.x 778 kB/s | 99 kB 00:00 2024-09-15T00:32:03.958 INFO:teuthology.orchestra.run.smithi059.stdout:(32/32): librados2-19.1.1-324.gecab7be5.el9.x86 9.5 MB/s | 3.4 MB 00:00 2024-09-15T00:32:03.964 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:32:03.964 INFO:teuthology.orchestra.run.smithi059.stdout:Total 13 MB/s | 60 MB 00:04 2024-09-15T00:32:03.968 INFO:teuthology.orchestra.run.smithi002.stdout:(13/32): python3-rbd-19.1.1-324.gecab7be5.el9.x 1.7 MB/s | 301 kB 00:00 2024-09-15T00:32:04.072 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:32:04.076 INFO:teuthology.orchestra.run.smithi002.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 419 kB/s | 45 kB 00:00 2024-09-15T00:32:04.102 INFO:teuthology.orchestra.run.smithi002.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 581 kB/s | 107 kB 00:00 2024-09-15T00:32:04.113 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:32:04.113 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:32:04.127 INFO:teuthology.orchestra.run.smithi002.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 870 kB/s | 160 kB 00:00 2024-09-15T00:32:04.177 INFO:teuthology.orchestra.run.smithi002.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-09-15T00:32:04.244 INFO:teuthology.orchestra.run.smithi002.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-15T00:32:04.269 INFO:teuthology.orchestra.run.smithi002.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-09-15T00:32:04.303 INFO:teuthology.orchestra.run.smithi002.stdout:(20/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 662 kB 00:00 2024-09-15T00:32:04.328 INFO:teuthology.orchestra.run.smithi002.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 428 kB/s | 36 kB 00:00 2024-09-15T00:32:04.345 INFO:teuthology.orchestra.run.smithi002.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-15T00:32:04.404 INFO:teuthology.orchestra.run.smithi002.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-15T00:32:04.421 INFO:teuthology.orchestra.run.smithi002.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-09-15T00:32:04.438 INFO:teuthology.orchestra.run.smithi002.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-15T00:32:04.463 INFO:teuthology.orchestra.run.smithi002.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-15T00:32:04.505 INFO:teuthology.orchestra.run.smithi002.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-15T00:32:04.539 INFO:teuthology.orchestra.run.smithi002.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-15T00:32:04.667 INFO:teuthology.orchestra.run.smithi002.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 9.8 MB/s | 1.6 MB 00:00 2024-09-15T00:32:04.781 INFO:teuthology.orchestra.run.smithi002.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-15T00:32:04.834 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:32:04.834 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:32:04.848 INFO:teuthology.orchestra.run.smithi002.stdout:(31/32): librados2-19.1.1-324.gecab7be5.el9.x86 11 MB/s | 3.4 MB 00:00 2024-09-15T00:32:04.931 INFO:teuthology.orchestra.run.smithi002.stdout:(32/32): librbd1-19.1.1-324.gecab7be5.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-15T00:32:04.937 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:32:04.937 INFO:teuthology.orchestra.run.smithi002.stdout:Total 22 MB/s | 60 MB 00:02 2024-09-15T00:32:05.050 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:32:05.091 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:32:05.092 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:32:05.824 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:32:05.824 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:32:06.454 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:32:06.546 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-15T00:32:06.579 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-15T00:32:06.605 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-15T00:32:06.636 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-15T00:32:06.657 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-15T00:32:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-15T00:32:07.025 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-15T00:32:07.377 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-15T00:32:07.379 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:32:07.463 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-15T00:32:07.481 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-15T00:32:07.495 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-15T00:32:07.520 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-15T00:32:07.543 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-15T00:32:07.571 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-15T00:32:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-15T00:32:07.680 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 8/34 2024-09-15T00:32:07.699 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 9/34 2024-09-15T00:32:07.762 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 9/34 2024-09-15T00:32:07.763 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-15T00:32:07.791 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-15T00:32:07.817 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-15T00:32:07.826 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x 11/34 2024-09-15T00:32:07.897 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 12/34 2024-09-15T00:32:07.926 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-15T00:32:07.990 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-15T00:32:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-15T00:32:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-15T00:32:08.170 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-15T00:32:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-15T00:32:08.222 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-15T00:32:08.249 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-15T00:32:08.278 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-15T00:32:08.280 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librados2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-15T00:32:08.311 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-15T00:32:08.356 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-15T00:32:08.446 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 8/34 2024-09-15T00:32:08.473 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 9/34 2024-09-15T00:32:08.519 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 9/34 2024-09-15T00:32:08.556 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-15T00:32:08.600 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x 11/34 2024-09-15T00:32:08.638 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-15T00:32:08.674 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 12/34 2024-09-15T00:32:08.682 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 22/34 2024-09-15T00:32:08.700 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-15T00:32:08.765 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-15T00:32:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 22/34 2024-09-15T00:32:08.799 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-15T00:32:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 23/34 2024-09-15T00:32:08.857 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-15T00:32:08.920 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-15T00:32:08.950 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-15T00:32:08.996 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-15T00:32:09.023 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-15T00:32:09.061 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-15T00:32:09.103 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-15T00:32:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-15T00:32:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 26/34 2024-09-15T00:32:09.437 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-15T00:32:09.464 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 26/34 2024-09-15T00:32:09.489 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 22/34 2024-09-15T00:32:09.546 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 27/34 2024-09-15T00:32:09.574 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 22/34 2024-09-15T00:32:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86 28/34 2024-09-15T00:32:09.603 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 23/34 2024-09-15T00:32:09.648 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-15T00:32:10.154 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-15T00:32:10.184 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 26/34 2024-09-15T00:32:10.255 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 26/34 2024-09-15T00:32:10.336 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 27/34 2024-09-15T00:32:10.361 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86 28/34 2024-09-15T00:32:12.001 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-15T00:32:12.049 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-15T00:32:12.677 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-15T00:32:12.710 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 30/34 2024-09-15T00:32:12.748 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-15T00:32:12.787 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-15T00:32:12.810 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 30/34 2024-09-15T00:32:12.810 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T00:32:12.810 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T00:32:12.811 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:12.862 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-15T00:32:13.417 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-15T00:32:13.453 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 30/34 2024-09-15T00:32:13.544 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 30/34 2024-09-15T00:32:13.544 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T00:32:13.544 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T00:32:13.544 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:13.587 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-15T00:32:23.360 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-15T00:32:23.360 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2024-09-15T00:32:23.360 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2024-09-15T00:32:23.361 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2024-09-15T00:32:23.361 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2024-09-15T00:32:23.361 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2024-09-15T00:32:23.361 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /root 2024-09-15T00:32:23.361 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2024-09-15T00:32:23.361 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:23.876 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-15T00:32:23.876 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /sys 2024-09-15T00:32:23.876 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /proc 2024-09-15T00:32:23.877 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /mnt 2024-09-15T00:32:23.877 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /var/tmp 2024-09-15T00:32:23.877 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /home 2024-09-15T00:32:23.877 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /root 2024-09-15T00:32:23.877 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /tmp 2024-09-15T00:32:23.877 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:24.358 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 32/34 2024-09-15T00:32:24.385 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 32/34 2024-09-15T00:32:24.385 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:32:24.385 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T00:32:24.385 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T00:32:24.386 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T00:32:24.386 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:24.387 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-15T00:32:24.468 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-15T00:32:24.469 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-15T00:32:24.879 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 32/34 2024-09-15T00:32:24.907 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 32/34 2024-09-15T00:32:24.907 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:32:24.907 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T00:32:24.907 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T00:32:24.907 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T00:32:24.907 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:24.909 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-15T00:32:24.998 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-15T00:32:24.998 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-15T00:32:25.721 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-15T00:32:25.721 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 1/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 3/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 4/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 5/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 6/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x 8/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86 9/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 10/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 11/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 12/34 2024-09-15T00:32:25.722 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-15T00:32:25.723 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-15T00:32:25.724 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-15T00:32:25.724 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-15T00:32:25.724 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-15T00:32:25.724 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-15T00:32:25.724 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-15T00:32:25.724 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 33/34 2024-09-15T00:32:26.188 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-15T00:32:26.188 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 1/34 2024-09-15T00:32:26.188 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2/34 2024-09-15T00:32:26.188 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 3/34 2024-09-15T00:32:26.188 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 4/34 2024-09-15T00:32:26.188 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 5/34 2024-09-15T00:32:26.188 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 6/34 2024-09-15T00:32:26.188 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-15T00:32:26.189 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x 8/34 2024-09-15T00:32:26.189 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86 9/34 2024-09-15T00:32:26.189 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 10/34 2024-09-15T00:32:26.189 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 11/34 2024-09-15T00:32:26.189 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 12/34 2024-09-15T00:32:26.189 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-15T00:32:26.189 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-15T00:32:26.189 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-15T00:32:26.190 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-15T00:32:26.191 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-15T00:32:26.191 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-15T00:32:26.191 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-15T00:32:26.191 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-15T00:32:26.191 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 33/34 2024-09-15T00:32:26.660 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-15T00:32:26.660 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:26.660 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2024-09-15T00:32:26.661 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.661 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.661 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:32:26.661 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T00:32:26.661 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.661 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.661 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.661 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.661 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T00:32:26.662 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T00:32:26.663 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:32:27.009 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2024-09-15T00:32:27.406 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-15T00:32:27.406 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:27.406 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2024-09-15T00:32:27.406 INFO:teuthology.orchestra.run.smithi002.stdout: librados2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T00:32:27.407 INFO:teuthology.orchestra.run.smithi002.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.408 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.409 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:27.409 INFO:teuthology.orchestra.run.smithi002.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T00:32:27.409 INFO:teuthology.orchestra.run.smithi002.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T00:32:27.409 INFO:teuthology.orchestra.run.smithi002.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T00:32:27.409 INFO:teuthology.orchestra.run.smithi002.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T00:32:27.409 INFO:teuthology.orchestra.run.smithi002.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T00:32:27.409 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:27.409 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:32:27.601 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:31 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:32:27.747 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:32:27.748 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:32:27.748 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-15T00:32:27.748 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:32:27.748 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:32:27.748 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:19.1.1-324.gecab7be5.el9 ceph 49 M 2024-09-15T00:32:27.748 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-15T00:32:27.748 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:19.1.1-324.gecab7be5.el9 ceph 163 k 2024-09-15T00:32:27.748 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T00:32:27.748 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T00:32:27.748 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:27.749 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:32:27.749 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:32:27.749 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2024-09-15T00:32:27.749 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:27.749 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 49 M 2024-09-15T00:32:27.749 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 209 M 2024-09-15T00:32:27.750 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:32:27.788 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-test 2024-09-15T00:32:27.983 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-15T00:32:28.033 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-15T00:32:28.058 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): libcephsqlite-19.1.1-324.gecab7be5.el9.x 570 kB/s | 163 kB 00:00 2024-09-15T00:32:28.377 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:29 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:32:28.517 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:32:28.517 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:28.517 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test x86_64 2:19.1.1-324.gecab7be5.el9 ceph 49 M 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite x86_64 2:19.1.1-324.gecab7be5.el9 ceph 163 k 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout:Install 4 Packages 2024-09-15T00:32:28.518 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:28.519 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 49 M 2024-09-15T00:32:28.519 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 209 M 2024-09-15T00:32:28.519 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:32:29.037 INFO:teuthology.orchestra.run.smithi002.stdout:(1/4): libcephsqlite-19.1.1-324.gecab7be5.el9.x 393 kB/s | 163 kB 00:00 2024-09-15T00:32:29.178 INFO:teuthology.orchestra.run.smithi002.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 546 kB/s | 304 kB 00:00 2024-09-15T00:32:29.403 INFO:teuthology.orchestra.run.smithi002.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 174 kB/s | 64 kB 00:00 2024-09-15T00:32:31.904 INFO:teuthology.orchestra.run.smithi002.stdout:(4/4): ceph-test-19.1.1-324.gecab7be5.el9.x86_6 15 MB/s | 49 MB 00:03 2024-09-15T00:32:31.906 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:32:31.906 INFO:teuthology.orchestra.run.smithi002.stdout:Total 15 MB/s | 49 MB 00:03 2024-09-15T00:32:31.934 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:32:31.956 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:32:31.956 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:32:32.385 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:32:32.386 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:32:32.938 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:32:33.011 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T00:32:33.060 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T00:32:33.089 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 3/4 2024-09-15T00:32:37.491 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): ceph-test-19.1.1-324.gecab7be5.el9.x86_6 5.0 MB/s | 49 MB 00:09 2024-09-15T00:32:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:32:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:Total 5.0 MB/s | 49 MB 00:09 2024-09-15T00:32:37.521 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:32:37.543 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:32:37.543 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:32:37.952 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 3/4 2024-09-15T00:32:37.957 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:32:37.957 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:32:38.040 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 4/4 2024-09-15T00:32:38.504 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:32:38.569 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T00:32:38.627 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T00:32:38.660 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 3/4 2024-09-15T00:32:40.575 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 4/4 2024-09-15T00:32:40.575 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 1/4 2024-09-15T00:32:40.575 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 2/4 2024-09-15T00:32:40.575 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T00:32:40.885 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T00:32:40.885 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:40.885 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:32:40.885 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:40.885 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:40.886 INFO:teuthology.orchestra.run.smithi002.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T00:32:40.886 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T00:32:40.886 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:40.886 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:32:41.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph 2024-09-15T00:32:41.790 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:42 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:32:41.934 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:32:41.937 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:41.937 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-09-15T00:32:41.937 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:41.937 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:32:41.937 INFO:teuthology.orchestra.run.smithi002.stdout: ceph x86_64 2:19.1.1-324.gecab7be5.el9 ceph 6.5 k 2024-09-15T00:32:41.937 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-09-15T00:32:41.937 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds x86_64 2:19.1.1-324.gecab7be5.el9 ceph 2.4 M 2024-09-15T00:32:41.937 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr x86_64 2:19.1.1-324.gecab7be5.el9 ceph 1.5 M 2024-09-15T00:32:41.937 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 249 k 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon x86_64 2:19.1.1-324.gecab7be5.el9 ceph 4.7 M 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 17 M 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T00:32:41.938 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T00:32:41.939 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout:Install 37 Packages 2024-09-15T00:32:41.940 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:41.941 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 30 M 2024-09-15T00:32:41.941 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 108 M 2024-09-15T00:32:41.941 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:32:42.579 INFO:teuthology.orchestra.run.smithi002.stdout:(1/37): ceph-19.1.1-324.gecab7be5.el9.x86_64.rp 20 kB/s | 6.5 kB 00:00 2024-09-15T00:32:43.441 INFO:teuthology.orchestra.run.smithi002.stdout:(2/37): ceph-mds-19.1.1-324.gecab7be5.el9.x86_6 2.0 MB/s | 2.4 MB 00:01 2024-09-15T00:32:43.483 INFO:teuthology.orchestra.run.smithi002.stdout:(3/37): ceph-mgr-19.1.1-324.gecab7be5.el9.x86_6 1.2 MB/s | 1.5 MB 00:01 2024-09-15T00:32:43.531 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 3/4 2024-09-15T00:32:43.611 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 4/4 2024-09-15T00:32:43.650 INFO:teuthology.orchestra.run.smithi002.stdout:(4/37): ceph-mgr-modules-core-19.1.1-324.gecab7 1.5 MB/s | 249 kB 00:00 2024-09-15T00:32:43.777 INFO:teuthology.orchestra.run.smithi002.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-09-15T00:32:43.884 INFO:teuthology.orchestra.run.smithi002.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-09-15T00:32:43.926 INFO:teuthology.orchestra.run.smithi002.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-15T00:32:43.968 INFO:teuthology.orchestra.run.smithi002.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-15T00:32:44.018 INFO:teuthology.orchestra.run.smithi002.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-15T00:32:44.060 INFO:teuthology.orchestra.run.smithi002.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 5.1 MB/s | 215 kB 00:00 2024-09-15T00:32:44.219 INFO:teuthology.orchestra.run.smithi002.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-15T00:32:44.286 INFO:teuthology.orchestra.run.smithi002.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-15T00:32:44.336 INFO:teuthology.orchestra.run.smithi002.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 698 kB/s | 35 kB 00:00 2024-09-15T00:32:44.386 INFO:teuthology.orchestra.run.smithi002.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-15T00:32:44.436 INFO:teuthology.orchestra.run.smithi002.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-15T00:32:44.487 INFO:teuthology.orchestra.run.smithi002.stdout:(16/37): python3-toml-0.10.2-6.el9.noarch.rpm 834 kB/s | 42 kB 00:00 2024-09-15T00:32:45.020 INFO:teuthology.orchestra.run.smithi002.stdout:(17/37): lua-devel-5.4.4-4.el9.x86_64.rpm 42 kB/s | 22 kB 00:00 2024-09-15T00:32:45.071 INFO:teuthology.orchestra.run.smithi002.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-15T00:32:45.096 INFO:teuthology.orchestra.run.smithi002.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-15T00:32:45.255 INFO:teuthology.orchestra.run.smithi002.stdout:(20/37): ceph-osd-19.1.1-324.gecab7be5.el9.x86_ 9.3 MB/s | 17 MB 00:01 2024-09-15T00:32:45.305 INFO:teuthology.orchestra.run.smithi002.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 823 kB/s | 172 kB 00:00 2024-09-15T00:32:45.330 INFO:teuthology.orchestra.run.smithi002.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 471 kB/s | 11 kB 00:00 2024-09-15T00:32:45.355 INFO:teuthology.orchestra.run.smithi002.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 710 kB/s | 18 kB 00:00 2024-09-15T00:32:45.380 INFO:teuthology.orchestra.run.smithi002.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 934 kB/s | 23 kB 00:00 2024-09-15T00:32:45.456 INFO:teuthology.orchestra.run.smithi002.stdout:(25/37): ceph-mon-19.1.1-324.gecab7be5.el9.x86_ 1.6 MB/s | 4.7 MB 00:02 2024-09-15T00:32:45.481 INFO:teuthology.orchestra.run.smithi002.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-09-15T00:32:45.515 INFO:teuthology.orchestra.run.smithi002.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-15T00:32:45.549 INFO:teuthology.orchestra.run.smithi002.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-15T00:32:45.574 INFO:teuthology.orchestra.run.smithi002.stdout:(29/37): python3-jaraco-text-3.2.0-6.el9.noarch 167 kB/s | 20 kB 00:00 2024-09-15T00:32:45.599 INFO:teuthology.orchestra.run.smithi002.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 961 kB/s | 79 kB 00:00 2024-09-15T00:32:45.625 INFO:teuthology.orchestra.run.smithi002.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-15T00:32:45.650 INFO:teuthology.orchestra.run.smithi002.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-15T00:32:45.675 INFO:teuthology.orchestra.run.smithi002.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-09-15T00:32:45.709 INFO:teuthology.orchestra.run.smithi002.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-15T00:32:45.734 INFO:teuthology.orchestra.run.smithi002.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 808 kB/s | 20 kB 00:00 2024-09-15T00:32:45.759 INFO:teuthology.orchestra.run.smithi002.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-15T00:32:45.784 INFO:teuthology.orchestra.run.smithi002.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-15T00:32:45.790 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:32:45.790 INFO:teuthology.orchestra.run.smithi002.stdout:Total 7.9 MB/s | 30 MB 00:03 2024-09-15T00:32:45.886 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 4/4 2024-09-15T00:32:45.886 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 1/4 2024-09-15T00:32:45.886 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 2/4 2024-09-15T00:32:45.886 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T00:32:45.948 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:32:45.992 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:32:45.992 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:32:46.208 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T00:32:46.208 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:46.208 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:32:46.208 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:46.208 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:46.208 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T00:32:46.208 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T00:32:46.208 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:46.209 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:32:46.487 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:32:46.488 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:32:46.520 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2024-09-15T00:32:47.106 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:32:47.249 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:19.1.1-324.gecab7be5.el9 ceph 6.5 k 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:19.1.1-324.gecab7be5.el9 ceph 2.4 M 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:19.1.1-324.gecab7be5.el9 ceph 1.5 M 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 249 k 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:19.1.1-324.gecab7be5.el9 ceph 4.7 M 2024-09-15T00:32:47.252 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 17 M 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T00:32:47.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T00:32:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T00:32:47.255 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T00:32:47.255 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T00:32:47.255 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T00:32:47.255 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T00:32:47.255 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T00:32:47.255 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:47.255 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:32:47.255 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:32:47.255 INFO:teuthology.orchestra.run.smithi059.stdout:Install 37 Packages 2024-09-15T00:32:47.255 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:47.256 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 30 M 2024-09-15T00:32:47.256 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 108 M 2024-09-15T00:32:47.256 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:32:47.608 INFO:teuthology.orchestra.run.smithi059.stdout:(1/37): ceph-19.1.1-324.gecab7be5.el9.x86_64.rp 50 kB/s | 6.5 kB 00:00 2024-09-15T00:32:47.800 INFO:teuthology.orchestra.run.smithi059.stdout:(2/37): ceph-mds-19.1.1-324.gecab7be5.el9.x86_6 7.4 MB/s | 2.4 MB 00:00 2024-09-15T00:32:47.841 INFO:teuthology.orchestra.run.smithi059.stdout:(3/37): ceph-mgr-19.1.1-324.gecab7be5.el9.x86_6 4.2 MB/s | 1.5 MB 00:00 2024-09-15T00:32:47.906 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:32:47.942 INFO:teuthology.orchestra.run.smithi059.stdout:(4/37): ceph-mgr-modules-core-19.1.1-324.gecab7 2.4 MB/s | 249 kB 00:00 2024-09-15T00:32:47.979 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-15T00:32:48.009 INFO:teuthology.orchestra.run.smithi059.stdout:(5/37): ceph-mon-19.1.1-324.gecab7be5.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-15T00:32:48.021 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-15T00:32:48.056 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-15T00:32:48.109 INFO:teuthology.orchestra.run.smithi059.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-15T00:32:48.150 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-15T00:32:48.151 INFO:teuthology.orchestra.run.smithi059.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-15T00:32:48.193 INFO:teuthology.orchestra.run.smithi059.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 6.7 MB/s | 1.2 MB 00:00 2024-09-15T00:32:48.197 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-15T00:32:48.218 INFO:teuthology.orchestra.run.smithi059.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-15T00:32:48.230 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-15T00:32:48.244 INFO:teuthology.orchestra.run.smithi059.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-15T00:32:48.269 INFO:teuthology.orchestra.run.smithi059.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-15T00:32:48.327 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-15T00:32:48.404 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-15T00:32:48.436 INFO:teuthology.orchestra.run.smithi059.stdout:(12/37): ceph-osd-19.1.1-324.gecab7be5.el9.x86_ 27 MB/s | 17 MB 00:00 2024-09-15T00:32:48.450 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-15T00:32:48.493 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-15T00:32:48.539 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-15T00:32:48.574 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-15T00:32:48.607 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-15T00:32:48.657 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-15T00:32:48.700 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-15T00:32:48.719 INFO:teuthology.orchestra.run.smithi059.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 123 kB/s | 35 kB 00:00 2024-09-15T00:32:48.734 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-15T00:32:48.784 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-15T00:32:48.811 INFO:teuthology.orchestra.run.smithi059.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 318 kB/s | 172 kB 00:00 2024-09-15T00:32:48.824 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-15T00:32:48.895 INFO:teuthology.orchestra.run.smithi059.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 289 kB/s | 188 kB 00:00 2024-09-15T00:32:48.900 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-15T00:32:48.920 INFO:teuthology.orchestra.run.smithi059.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 385 kB/s | 77 kB 00:00 2024-09-15T00:32:48.945 INFO:teuthology.orchestra.run.smithi059.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 381 kB/s | 51 kB 00:00 2024-09-15T00:32:48.971 INFO:teuthology.orchestra.run.smithi059.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 444 kB/s | 22 kB 00:00 2024-09-15T00:32:49.012 INFO:teuthology.orchestra.run.smithi059.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 355 kB/s | 42 kB 00:00 2024-09-15T00:32:49.046 INFO:teuthology.orchestra.run.smithi059.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 577 kB/s | 43 kB 00:00 2024-09-15T00:32:49.070 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-15T00:32:49.071 INFO:teuthology.orchestra.run.smithi059.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-09-15T00:32:49.097 INFO:teuthology.orchestra.run.smithi059.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 432 kB/s | 11 kB 00:00 2024-09-15T00:32:49.122 INFO:teuthology.orchestra.run.smithi059.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-15T00:32:49.150 INFO:teuthology.orchestra.run.smithi059.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 332 kB/s | 18 kB 00:00 2024-09-15T00:32:49.160 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-15T00:32:49.175 INFO:teuthology.orchestra.run.smithi059.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 435 kB/s | 23 kB 00:00 2024-09-15T00:32:49.201 INFO:teuthology.orchestra.run.smithi059.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-15T00:32:49.219 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-15T00:32:49.226 INFO:teuthology.orchestra.run.smithi059.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-09-15T00:32:49.251 INFO:teuthology.orchestra.run.smithi059.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-09-15T00:32:49.256 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-15T00:32:49.277 INFO:teuthology.orchestra.run.smithi059.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-15T00:32:49.302 INFO:teuthology.orchestra.run.smithi059.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-15T00:32:49.315 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-15T00:32:49.327 INFO:teuthology.orchestra.run.smithi059.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-15T00:32:49.352 INFO:teuthology.orchestra.run.smithi059.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-15T00:32:49.378 INFO:teuthology.orchestra.run.smithi059.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-15T00:32:49.379 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-15T00:32:49.403 INFO:teuthology.orchestra.run.smithi059.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-15T00:32:49.428 INFO:teuthology.orchestra.run.smithi059.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 796 kB/s | 20 kB 00:00 2024-09-15T00:32:49.453 INFO:teuthology.orchestra.run.smithi059.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-15T00:32:49.479 INFO:teuthology.orchestra.run.smithi059.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-15T00:32:49.483 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:32:49.483 INFO:teuthology.orchestra.run.smithi059.stdout:Total 14 MB/s | 30 MB 00:02 2024-09-15T00:32:49.584 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-15T00:32:49.635 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:32:49.636 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-15T00:32:49.679 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:32:49.679 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:32:49.717 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-15T00:32:49.815 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-15T00:32:49.851 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-15T00:32:49.983 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-15T00:32:50.155 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.n 32/37 2024-09-15T00:32:50.164 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:32:50.164 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:32:50.186 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 33/37 2024-09-15T00:32:50.214 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 33/37 2024-09-15T00:32:50.214 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:32:50.214 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T00:32:50.214 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T00:32:50.214 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T00:32:50.214 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:51.534 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:32:51.590 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-15T00:32:51.630 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-15T00:32:51.659 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-15T00:32:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-15T00:32:51.737 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 34/37 2024-09-15T00:32:51.767 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 34/37 2024-09-15T00:32:51.767 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:32:51.767 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T00:32:51.767 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T00:32:51.767 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T00:32:51.767 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-15T00:32:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-15T00:32:51.911 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-15T00:32:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-15T00:32:52.044 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-15T00:32:52.079 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-15T00:32:52.108 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-15T00:32:52.143 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-15T00:32:52.176 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-15T00:32:52.217 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-15T00:32:52.237 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 35/37 2024-09-15T00:32:52.260 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-15T00:32:52.263 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 35/37 2024-09-15T00:32:52.263 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:32:52.263 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T00:32:52.263 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T00:32:52.263 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T00:32:52.263 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-15T00:32:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-15T00:32:52.384 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-15T00:32:52.460 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-15T00:32:52.505 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 36/37 2024-09-15T00:32:52.530 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 36/37 2024-09-15T00:32:52.530 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:32:52.530 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T00:32:52.531 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T00:32:52.531 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T00:32:52.531 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:52.596 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-2:19.1.1-324.gecab7be5.el9.x86_64 37/37 2024-09-15T00:32:52.613 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-15T00:32:52.701 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-15T00:32:52.752 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-15T00:32:52.915 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-15T00:32:52.974 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-15T00:32:53.037 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-15T00:32:53.232 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-15T00:32:53.285 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-15T00:32:53.367 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-15T00:32:53.465 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-15T00:32:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-15T00:32:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-15T00:32:53.788 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.n 32/37 2024-09-15T00:32:53.819 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 33/37 2024-09-15T00:32:53.846 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 33/37 2024-09-15T00:32:53.846 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:32:53.846 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T00:32:53.846 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T00:32:53.846 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T00:32:53.846 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:54.401 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-2:19.1.1-324.gecab7be5.el9.x86_64 37/37 2024-09-15T00:32:54.401 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-2:19.1.1-324.gecab7be5.el9.x86_64 1/37 2024-09-15T00:32:54.401 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 2/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 3/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 4/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 5/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.n 6/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-15T00:32:54.402 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-15T00:32:54.403 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-15T00:32:54.404 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-15T00:32:54.404 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-15T00:32:54.404 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-15T00:32:54.404 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-15T00:32:54.404 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-15T00:32:54.404 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-15T00:32:54.404 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-15T00:32:54.404 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-15T00:32:55.362 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 34/37 2024-09-15T00:32:55.390 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 34/37 2024-09-15T00:32:55.390 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:32:55.391 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T00:32:55.391 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T00:32:55.391 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T00:32:55.391 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:55.442 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-15T00:32:55.442 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:55.442 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:32:55.442 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:55.442 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:55.442 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T00:32:55.443 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T00:32:55.444 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T00:32:55.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T00:32:55.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T00:32:55.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T00:32:55.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T00:32:55.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T00:32:55.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T00:32:55.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T00:32:55.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T00:32:55.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T00:32:55.446 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T00:32:55.446 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:55.446 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:32:55.825 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-base 2024-09-15T00:32:55.871 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 35/37 2024-09-15T00:32:55.896 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 35/37 2024-09-15T00:32:55.896 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:32:55.897 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T00:32:55.897 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T00:32:55.897 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T00:32:55.897 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:56.138 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 36/37 2024-09-15T00:32:56.164 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 36/37 2024-09-15T00:32:56.164 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:32:56.164 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T00:32:56.164 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T00:32:56.164 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T00:32:56.164 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:56.238 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:19.1.1-324.gecab7be5.el9.x86_64 37/37 2024-09-15T00:32:56.420 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:32:56.513 INFO:teuthology.orchestra.run.smithi002.stdout:Package ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:32:56.558 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:32:56.559 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-09-15T00:32:56.559 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:32:56.641 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install cephadm 2024-09-15T00:32:57.231 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:58 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:32:57.369 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:32:57.370 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:57.370 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-09-15T00:32:57.370 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:57.370 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:32:57.370 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 765 k 2024-09-15T00:32:57.370 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:57.370 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:32:57.370 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:57.370 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-09-15T00:32:57.370 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:57.371 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 765 k 2024-09-15T00:32:57.371 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 771 k 2024-09-15T00:32:57.371 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:32:57.653 INFO:teuthology.orchestra.run.smithi002.stdout:cephadm-19.1.1-324.gecab7be5.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-15T00:32:57.654 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:32:57.654 INFO:teuthology.orchestra.run.smithi002.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-15T00:32:57.654 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:32:57.661 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:32:57.661 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:32:57.666 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:32:57.666 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:32:57.746 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:32:57.870 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:19.1.1-324.gecab7be5.el9.x86_64 37/37 2024-09-15T00:32:57.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:19.1.1-324.gecab7be5.el9.x86_64 1/37 2024-09-15T00:32:57.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 2/37 2024-09-15T00:32:57.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 3/37 2024-09-15T00:32:57.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 4/37 2024-09-15T00:32:57.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 5/37 2024-09-15T00:32:57.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.n 6/37 2024-09-15T00:32:57.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-15T00:32:57.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-15T00:32:57.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-15T00:32:57.871 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-15T00:32:57.872 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-15T00:32:58.208 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:32:58.318 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:32:58.694 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:32:58.933 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-15T00:32:58.933 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:58.933 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:32:58.933 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:58.933 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T00:32:58.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T00:32:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:32:58.936 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:32:58.948 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:32:58.948 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:58.948 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:32:58.948 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:32:58.949 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:58.949 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:32:59.201 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T00:32:59.303 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2024-09-15T00:32:59.792 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:00 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:32:59.898 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:03 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:32:59.931 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:32:59.932 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:59.932 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2024-09-15T00:32:59.932 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:59.932 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:32:59.932 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-324.gecab7be5.el9 ceph 145 k 2024-09-15T00:32:59.932 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:59.932 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:32:59.932 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:32:59.932 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-09-15T00:32:59.932 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:32:59.933 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 145 k 2024-09-15T00:32:59.933 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 435 k 2024-09-15T00:32:59.933 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:32:59.993 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:00.037 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:00.038 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-15T00:33:00.038 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:00.107 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2024-09-15T00:33:00.144 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-immutable-object-cache-19.1.1-324.gecab7be 687 kB/s | 145 kB 00:00 2024-09-15T00:33:00.145 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:00.145 INFO:teuthology.orchestra.run.smithi002.stdout:Total 682 kB/s | 145 kB 00:00 2024-09-15T00:33:00.145 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:00.153 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:00.153 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:00.213 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:00.214 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:00.360 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:00.445 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-15T00:33:00.471 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-15T00:33:00.471 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:33:00.471 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T00:33:00.471 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:00.693 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:04 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:00.832 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:00.833 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:00.833 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-15T00:33:00.833 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:00.834 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:00.834 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 765 k 2024-09-15T00:33:00.834 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:00.834 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:00.834 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:00.834 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-15T00:33:00.834 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:00.834 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 765 k 2024-09-15T00:33:00.834 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 771 k 2024-09-15T00:33:00.834 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:01.111 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-19.1.1-324.gecab7be5.el9.noarch.rpm 2.7 MB/s | 765 kB 00:00 2024-09-15T00:33:01.112 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:01.112 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.7 MB/s | 765 kB 00:00 2024-09-15T00:33:01.113 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:01.118 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:01.118 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:01.124 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:01.125 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:01.197 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-15T00:33:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:01.381 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:01.381 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:01.634 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr 2024-09-15T00:33:01.657 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:33:01.751 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:33:02.099 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:33:02.234 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:03 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:02.327 INFO:teuthology.orchestra.run.smithi002.stdout:Package ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:02.357 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:33:02.357 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:02.357 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:02.357 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:02.357 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:02.357 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:02.371 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:02.372 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-09-15T00:33:02.372 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:02.453 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T00:33:02.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T00:33:03.042 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:04 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:03.136 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:07 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:03.183 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-09-15T00:33:03.183 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:03.183 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 3.6 M 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 32 k 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 16 k 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T00:33:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T00:33:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T00:33:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:03.185 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:03.185 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:03.185 INFO:teuthology.orchestra.run.smithi002.stdout:Install 12 Packages 2024-09-15T00:33:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:03.185 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 8.7 M 2024-09-15T00:33:03.186 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 104 M 2024-09-15T00:33:03.186 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:03.274 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:03.274 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:03.274 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-09-15T00:33:03.274 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:03.274 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:03.274 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-324.gecab7be5.el9 ceph 145 k 2024-09-15T00:33:03.274 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:03.274 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:03.275 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:03.275 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-15T00:33:03.275 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:03.275 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 145 k 2024-09-15T00:33:03.275 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 435 k 2024-09-15T00:33:03.275 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:03.493 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-19.1.1-324.gecab7be 667 kB/s | 145 kB 00:00 2024-09-15T00:33:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:Total 662 kB/s | 145 kB 00:00 2024-09-15T00:33:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:03.502 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:03.502 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:03.563 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:03.564 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:03.656 INFO:teuthology.orchestra.run.smithi002.stdout:(1/12): ceph-grafana-dashboards-19.1.1-324.geca 232 kB/s | 32 kB 00:00 2024-09-15T00:33:03.681 INFO:teuthology.orchestra.run.smithi002.stdout:(2/12): ceph-prometheus-alerts-19.1.1-324.gecab 101 kB/s | 16 kB 00:00 2024-09-15T00:33:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-15T00:33:03.796 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-15T00:33:03.796 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:33:03.796 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T00:33:03.796 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:03.865 INFO:teuthology.orchestra.run.smithi002.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-09-15T00:33:03.965 INFO:teuthology.orchestra.run.smithi002.stdout:(4/12): ceph-mgr-dashboard-19.1.1-324.gecab7be5 8.0 MB/s | 3.6 MB 00:00 2024-09-15T00:33:04.015 INFO:teuthology.orchestra.run.smithi002.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-15T00:33:04.040 INFO:teuthology.orchestra.run.smithi002.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 776 kB/s | 19 kB 00:00 2024-09-15T00:33:04.074 INFO:teuthology.orchestra.run.smithi002.stdout:(7/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.4 MB/s | 1.0 MB 00:00 2024-09-15T00:33:04.124 INFO:teuthology.orchestra.run.smithi002.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-09-15T00:33:04.166 INFO:teuthology.orchestra.run.smithi002.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.5 MB/s | 144 kB 00:00 2024-09-15T00:33:04.183 INFO:teuthology.orchestra.run.smithi002.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 544 kB/s | 31 kB 00:00 2024-09-15T00:33:04.208 INFO:teuthology.orchestra.run.smithi002.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.6 MB/s | 188 kB 00:00 2024-09-15T00:33:04.259 INFO:teuthology.orchestra.run.smithi002.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.4 MB/s | 2.0 MB 00:00 2024-09-15T00:33:04.262 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:04.262 INFO:teuthology.orchestra.run.smithi002.stdout:Total 8.0 MB/s | 8.7 MB 00:01 2024-09-15T00:33:04.312 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:04.331 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:04.331 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:04.573 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:04.573 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:04.724 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-15T00:33:04.724 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:04.724 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:04.724 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:04.724 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:04.725 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:04.971 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2024-09-15T00:33:05.115 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:05.202 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-15T00:33:05.246 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-15T00:33:05.293 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-15T00:33:05.322 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-15T00:33:05.435 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-15T00:33:05.560 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:09 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:05.570 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-15T00:33:05.654 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:05.662 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-15T00:33:05.699 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:05.700 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-15T00:33:05.701 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:05.710 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-15T00:33:05.738 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-15T00:33:05.770 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9. 10/12 2024-09-15T00:33:05.771 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T00:33:06.361 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:10 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:06.499 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:06.500 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:06.500 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-15T00:33:06.500 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:06.500 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:06.500 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 3.6 M 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 32 k 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 16 k 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:06.501 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:06.502 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:06.502 INFO:teuthology.orchestra.run.smithi059.stdout:Install 12 Packages 2024-09-15T00:33:06.502 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:06.502 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 8.7 M 2024-09-15T00:33:06.502 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 104 M 2024-09-15T00:33:06.502 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9 11/12 2024-09-15T00:33:06.913 INFO:teuthology.orchestra.run.smithi059.stdout:(1/12): ceph-grafana-dashboards-19.1.1-324.geca 212 kB/s | 32 kB 00:00 2024-09-15T00:33:06.929 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 12/12 2024-09-15T00:33:06.929 INFO:teuthology.orchestra.run.smithi059.stdout:(2/12): ceph-prometheus-alerts-19.1.1-324.gecab 99 kB/s | 16 kB 00:00 2024-09-15T00:33:07.122 INFO:teuthology.orchestra.run.smithi059.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-09-15T00:33:07.180 INFO:teuthology.orchestra.run.smithi059.stdout:(4/12): ceph-mgr-dashboard-19.1.1-324.gecab7be5 8.5 MB/s | 3.6 MB 00:00 2024-09-15T00:33:07.230 INFO:teuthology.orchestra.run.smithi059.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-15T00:33:07.264 INFO:teuthology.orchestra.run.smithi059.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-09-15T00:33:07.281 INFO:teuthology.orchestra.run.smithi059.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 387 kB/s | 19 kB 00:00 2024-09-15T00:33:07.315 INFO:teuthology.orchestra.run.smithi059.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.4 MB/s | 144 kB 00:00 2024-09-15T00:33:07.331 INFO:teuthology.orchestra.run.smithi059.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 2.1 MB/s | 31 kB 00:00 2024-09-15T00:33:07.357 INFO:teuthology.orchestra.run.smithi059.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-15T00:33:07.390 INFO:teuthology.orchestra.run.smithi059.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 3.1 MB/s | 863 kB 00:00 2024-09-15T00:33:07.449 INFO:teuthology.orchestra.run.smithi059.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-15T00:33:07.453 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:07.453 INFO:teuthology.orchestra.run.smithi059.stdout:Total 9.1 MB/s | 8.7 MB 00:00 2024-09-15T00:33:07.503 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:07.521 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:07.521 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:07.734 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:07.734 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:08.293 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:08.348 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 12/12 2024-09-15T00:33:08.348 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9 1/12 2024-09-15T00:33:08.348 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 2/12 2024-09-15T00:33:08.348 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9. 3/12 2024-09-15T00:33:08.348 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-15T00:33:08.348 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-15T00:33:08.348 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-15T00:33:08.349 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-15T00:33:08.349 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-15T00:33:08.349 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-15T00:33:08.349 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-15T00:33:08.349 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-15T00:33:08.384 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-15T00:33:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-15T00:33:08.474 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-15T00:33:08.511 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-15T00:33:08.611 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-15T00:33:08.672 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-15T00:33:08.672 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:08.672 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:08.672 INFO:teuthology.orchestra.run.smithi002.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T00:33:08.673 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:08.674 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:08.743 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-15T00:33:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-15T00:33:08.892 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-15T00:33:08.904 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T00:33:08.919 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-15T00:33:08.951 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9. 10/12 2024-09-15T00:33:09.503 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:10 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:09.662 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:09.663 INFO:teuthology.orchestra.run.smithi002.stdout:====================================================================================== 2024-09-15T00:33:09.663 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-09-15T00:33:09.663 INFO:teuthology.orchestra.run.smithi002.stdout:====================================================================================== 2024-09-15T00:33:09.663 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:09.663 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 7.4 M 2024-09-15T00:33:09.663 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout:====================================================================================== 2024-09-15T00:33:09.664 INFO:teuthology.orchestra.run.smithi002.stdout:Install 12 Packages 2024-09-15T00:33:09.665 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:09.665 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 38 M 2024-09-15T00:33:09.665 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 212 M 2024-09-15T00:33:09.665 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:10.064 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9 11/12 2024-09-15T00:33:10.102 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 12/12 2024-09-15T00:33:10.501 INFO:teuthology.orchestra.run.smithi002.stdout:(1/12): ceph-mgr-diskprediction-local-19.1.1-32 14 MB/s | 7.4 MB 00:00 2024-09-15T00:33:10.529 INFO:teuthology.orchestra.run.smithi002.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 340 kB/s | 191 kB 00:00 2024-09-15T00:33:10.584 INFO:teuthology.orchestra.run.smithi002.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 390 kB/s | 33 kB 00:00 2024-09-15T00:33:10.618 INFO:teuthology.orchestra.run.smithi002.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 546 kB/s | 18 kB 00:00 2024-09-15T00:33:10.651 INFO:teuthology.orchestra.run.smithi002.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 1.1 MB/s | 38 kB 00:00 2024-09-15T00:33:10.710 INFO:teuthology.orchestra.run.smithi002.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 801 kB 00:00 2024-09-15T00:33:10.769 INFO:teuthology.orchestra.run.smithi002.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-09-15T00:33:10.937 INFO:teuthology.orchestra.run.smithi002.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 31 MB/s | 5.1 MB 00:00 2024-09-15T00:33:11.019 INFO:teuthology.orchestra.run.smithi002.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2024-09-15T00:33:11.045 INFO:teuthology.orchestra.run.smithi002.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 732 kB/s | 245 kB 00:00 2024-09-15T00:33:11.070 INFO:teuthology.orchestra.run.smithi002.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.8 MB/s | 250 kB 00:00 2024-09-15T00:33:11.395 INFO:teuthology.orchestra.run.smithi002.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-09-15T00:33:11.396 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:11.397 INFO:teuthology.orchestra.run.smithi002.stdout:Total 22 MB/s | 38 MB 00:01 2024-09-15T00:33:11.454 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 12/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9 1/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 2/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9. 3/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-15T00:33:11.455 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-15T00:33:11.763 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:11.790 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:11.790 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:11.799 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-15T00:33:11.799 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:11.799 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:11.799 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T00:33:11.799 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:11.799 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:11.799 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:11.799 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T00:33:11.800 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T00:33:11.800 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T00:33:11.800 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T00:33:11.800 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T00:33:11.800 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T00:33:11.800 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T00:33:11.800 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T00:33:11.800 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:11.800 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:12.021 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T00:33:12.244 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:12.245 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:12.618 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:16 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:12.758 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 7.4 M 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T00:33:12.759 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T00:33:12.760 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T00:33:12.760 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T00:33:12.760 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T00:33:12.760 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T00:33:12.760 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T00:33:12.760 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:12.760 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:12.760 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-09-15T00:33:12.760 INFO:teuthology.orchestra.run.smithi059.stdout:Install 12 Packages 2024-09-15T00:33:12.760 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:12.761 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 38 M 2024-09-15T00:33:12.761 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 212 M 2024-09-15T00:33:12.761 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:13.032 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:13.119 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T00:33:13.123 INFO:teuthology.orchestra.run.smithi059.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-15T00:33:13.163 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T00:33:13.174 INFO:teuthology.orchestra.run.smithi059.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 655 kB/s | 33 kB 00:00 2024-09-15T00:33:13.265 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T00:33:13.303 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T00:33:13.385 INFO:teuthology.orchestra.run.smithi059.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.8 MB/s | 801 kB 00:00 2024-09-15T00:33:13.474 INFO:teuthology.orchestra.run.smithi059.stdout:(4/12): ceph-mgr-diskprediction-local-19.1.1-32 14 MB/s | 7.4 MB 00:00 2024-09-15T00:33:13.495 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T00:33:13.499 INFO:teuthology.orchestra.run.smithi059.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 156 kB/s | 18 kB 00:00 2024-09-15T00:33:13.527 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T00:33:13.583 INFO:teuthology.orchestra.run.smithi059.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 355 kB/s | 38 kB 00:00 2024-09-15T00:33:13.614 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T00:33:13.691 INFO:teuthology.orchestra.run.smithi059.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-15T00:33:14.125 INFO:teuthology.orchestra.run.smithi059.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.1 MB/s | 3.0 MB 00:00 2024-09-15T00:33:14.234 INFO:teuthology.orchestra.run.smithi059.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-09-15T00:33:14.311 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T00:33:14.428 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T00:33:14.709 INFO:teuthology.orchestra.run.smithi059.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.0 MB/s | 5.1 MB 00:01 2024-09-15T00:33:14.917 INFO:teuthology.orchestra.run.smithi059.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.5 MB/s | 4.9 MB 00:01 2024-09-15T00:33:15.752 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T00:33:16.826 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T00:33:16.868 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 12/12 2024-09-15T00:33:16.917 INFO:teuthology.orchestra.run.smithi059.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.1 MB/s | 16 MB 00:02 2024-09-15T00:33:16.918 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:16.919 INFO:teuthology.orchestra.run.smithi059.stdout:Total 9.2 MB/s | 38 MB 00:04 2024-09-15T00:33:17.282 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:17.309 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:17.309 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:17.759 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:17.759 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:18.668 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:18.739 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T00:33:18.790 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T00:33:18.899 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T00:33:18.942 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T00:33:19.125 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T00:33:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T00:33:19.245 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T00:33:19.473 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 12/12 2024-09-15T00:33:19.473 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 1/12 2024-09-15T00:33:19.473 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T00:33:19.473 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T00:33:19.473 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T00:33:19.473 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T00:33:19.473 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T00:33:19.473 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T00:33:19.473 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T00:33:19.473 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T00:33:19.474 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T00:33:19.474 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T00:33:19.944 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T00:33:19.985 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T00:33:19.985 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:19.985 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:19.985 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:19.986 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T00:33:20.354 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-rook 2024-09-15T00:33:20.950 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:21 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:21.090 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:21.091 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:21.091 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-09-15T00:33:21.091 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:21.091 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:21.091 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 49 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T00:33:21.092 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T00:33:21.093 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:21.093 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:21.093 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:21.093 INFO:teuthology.orchestra.run.smithi002.stdout:Install 13 Packages 2024-09-15T00:33:21.093 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:21.093 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 2.2 M 2024-09-15T00:33:21.093 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 26 M 2024-09-15T00:33:21.094 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:21.285 INFO:teuthology.orchestra.run.smithi002.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 209 kB/s | 19 kB 00:00 2024-09-15T00:33:21.310 INFO:teuthology.orchestra.run.smithi002.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 219 kB/s | 26 kB 00:00 2024-09-15T00:33:21.377 INFO:teuthology.orchestra.run.smithi002.stdout:(3/13): ceph-mgr-rook-19.1.1-324.gecab7be5.el9. 268 kB/s | 49 kB 00:00 2024-09-15T00:33:21.396 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T00:33:21.411 INFO:teuthology.orchestra.run.smithi002.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-09-15T00:33:21.436 INFO:teuthology.orchestra.run.smithi002.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-15T00:33:21.461 INFO:teuthology.orchestra.run.smithi002.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-15T00:33:21.503 INFO:teuthology.orchestra.run.smithi002.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-15T00:33:21.529 INFO:teuthology.orchestra.run.smithi002.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-15T00:33:21.554 INFO:teuthology.orchestra.run.smithi002.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-15T00:33:21.588 INFO:teuthology.orchestra.run.smithi002.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-15T00:33:21.613 INFO:teuthology.orchestra.run.smithi002.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-15T00:33:21.638 INFO:teuthology.orchestra.run.smithi002.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 705 kB/s | 59 kB 00:00 2024-09-15T00:33:21.680 INFO:teuthology.orchestra.run.smithi002.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-15T00:33:21.683 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:21.683 INFO:teuthology.orchestra.run.smithi002.stdout:Total 3.8 MB/s | 2.2 MB 00:00 2024-09-15T00:33:21.750 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:21.767 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:21.767 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:21.867 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:21.868 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:22.394 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:22.478 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T00:33:22.506 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 12/12 2024-09-15T00:33:22.576 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T00:33:22.633 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T00:33:22.697 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T00:33:22.733 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T00:33:22.774 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T00:33:22.880 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T00:33:22.990 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T00:33:23.043 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T00:33:23.809 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T00:33:23.864 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T00:33:23.898 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T00:33:23.947 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T00:33:23.976 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 13/13 2024-09-15T00:33:24.750 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 13/13 2024-09-15T00:33:24.750 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 1/13 2024-09-15T00:33:24.750 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T00:33:24.750 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T00:33:24.750 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T00:33:24.750 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T00:33:24.751 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T00:33:24.751 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T00:33:24.751 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T00:33:24.751 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T00:33:24.751 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T00:33:24.751 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T00:33:24.751 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T00:33:24.966 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 12/12 2024-09-15T00:33:24.966 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 1/12 2024-09-15T00:33:24.966 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T00:33:24.966 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T00:33:24.967 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T00:33:24.967 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T00:33:24.967 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T00:33:24.967 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T00:33:24.967 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T00:33:24.967 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T00:33:24.967 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T00:33:24.967 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T00:33:25.080 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T00:33:25.081 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:25.081 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:25.081 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:25.081 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T00:33:25.081 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T00:33:25.081 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T00:33:25.081 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T00:33:25.081 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T00:33:25.081 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T00:33:25.082 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T00:33:25.082 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T00:33:25.082 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T00:33:25.082 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T00:33:25.082 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T00:33:25.082 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T00:33:25.082 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:25.083 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:25.362 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T00:33:25.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T00:33:25.381 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:25.381 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:25.381 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:25.381 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T00:33:25.381 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:25.382 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:25.777 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2024-09-15T00:33:25.956 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:26.096 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:26.097 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:26.097 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-09-15T00:33:26.097 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:26.097 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:26.097 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 146 k 2024-09-15T00:33:26.097 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-09-15T00:33:26.097 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T00:33:26.098 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T00:33:26.098 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T00:33:26.098 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T00:33:26.098 INFO:teuthology.orchestra.run.smithi002.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T00:33:26.098 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:26.098 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:26.098 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:26.098 INFO:teuthology.orchestra.run.smithi002.stdout:Install 6 Packages 2024-09-15T00:33:26.098 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:26.099 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 7.0 M 2024-09-15T00:33:26.099 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 33 M 2024-09-15T00:33:26.099 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:26.365 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:30 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-15T00:33:26.506 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:26.507 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:26.507 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 49 k 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T00:33:26.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T00:33:26.509 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T00:33:26.509 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T00:33:26.509 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T00:33:26.509 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T00:33:26.509 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:26.509 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:26.509 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:26.509 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2024-09-15T00:33:26.509 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:26.510 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.2 M 2024-09-15T00:33:26.510 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 26 M 2024-09-15T00:33:26.510 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:(2/6): ceph-mgr-cephadm-19.1.1-324.gecab7be5.el 640 kB/s | 146 kB 00:00 2024-09-15T00:33:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2024-09-15T00:33:26.582 INFO:teuthology.orchestra.run.smithi002.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.8 MB/s | 548 kB 00:00 2024-09-15T00:33:26.607 INFO:teuthology.orchestra.run.smithi002.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-15T00:33:26.754 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 223 kB/s | 26 kB 00:00 2024-09-15T00:33:26.779 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 137 kB/s | 19 kB 00:00 2024-09-15T00:33:26.805 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): ceph-mgr-rook-19.1.1-324.gecab7be5.el9. 298 kB/s | 49 kB 00:00 2024-09-15T00:33:26.874 INFO:teuthology.orchestra.run.smithi002.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-09-15T00:33:26.877 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:26.877 INFO:teuthology.orchestra.run.smithi002.stdout:Total 9.0 MB/s | 7.0 MB 00:00 2024-09-15T00:33:26.922 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-15T00:33:26.947 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 951 kB/s | 159 kB 00:00 2024-09-15T00:33:26.951 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:26.963 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:26.963 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:26.989 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 776 kB/s | 32 kB 00:00 2024-09-15T00:33:27.014 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 802 kB/s | 54 kB 00:00 2024-09-15T00:33:27.039 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 281 kB/s | 14 kB 00:00 2024-09-15T00:33:27.070 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:27.071 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:27.073 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.0 MB/s | 279 kB 00:00 2024-09-15T00:33:27.115 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.1 MB/s | 220 kB 00:00 2024-09-15T00:33:27.140 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 910 kB/s | 59 kB 00:00 2024-09-15T00:33:27.165 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-15T00:33:27.199 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-09-15T00:33:27.201 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:27.202 INFO:teuthology.orchestra.run.smithi059.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-15T00:33:27.271 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:27.287 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:27.288 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:27.388 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:27.388 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:27.392 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:27.568 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T00:33:27.620 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T00:33:28.031 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:28.233 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T00:33:28.233 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T00:33:28.268 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T00:33:28.291 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T00:33:28.325 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T00:33:28.360 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T00:33:28.363 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T00:33:28.388 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 6/6 2024-09-15T00:33:28.401 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T00:33:28.508 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T00:33:28.638 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T00:33:28.695 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T00:33:29.164 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 6/6 2024-09-15T00:33:29.164 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/6 2024-09-15T00:33:29.164 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T00:33:29.164 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T00:33:29.164 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T00:33:29.164 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T00:33:29.445 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T00:33:29.445 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:29.445 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:29.445 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:29.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T00:33:29.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T00:33:29.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T00:33:29.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T00:33:29.445 INFO:teuthology.orchestra.run.smithi002.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T00:33:29.445 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:29.446 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:29.453 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T00:33:29.508 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T00:33:29.541 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T00:33:29.599 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T00:33:29.644 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 13/13 2024-09-15T00:33:29.706 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-fuse 2024-09-15T00:33:30.316 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:30.426 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 13/13 2024-09-15T00:33:30.426 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 1/13 2024-09-15T00:33:30.426 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T00:33:30.426 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T00:33:30.426 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T00:33:30.426 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T00:33:30.426 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T00:33:30.426 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T00:33:30.426 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T00:33:30.427 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T00:33:30.427 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T00:33:30.427 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T00:33:30.427 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T00:33:30.456 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-09-15T00:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 896 k 2024-09-15T00:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-09-15T00:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T00:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout:Install 2 Packages 2024-09-15T00:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 976 k 2024-09-15T00:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 2.9 M 2024-09-15T00:33:30.459 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:30.660 INFO:teuthology.orchestra.run.smithi002.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 523 kB/s | 80 kB 00:00 2024-09-15T00:33:30.763 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T00:33:30.764 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T00:33:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T00:33:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T00:33:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T00:33:30.765 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:30.765 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:30.818 INFO:teuthology.orchestra.run.smithi002.stdout:(2/2): ceph-fuse-19.1.1-324.gecab7be5.el9.x86_6 2.8 MB/s | 896 kB 00:00 2024-09-15T00:33:30.819 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:30.819 INFO:teuthology.orchestra.run.smithi002.stdout:Total 2.6 MB/s | 976 kB 00:00 2024-09-15T00:33:30.831 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:30.855 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:30.855 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:30.941 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:30.942 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:30.996 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T00:33:31.144 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:31.291 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T00:33:31.362 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2/2 2024-09-15T00:33:31.585 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:31.725 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:31.725 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:31.725 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-15T00:33:31.725 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 146 k 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-09-15T00:33:31.726 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:31.727 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 7.0 M 2024-09-15T00:33:31.727 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 33 M 2024-09-15T00:33:31.727 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:32.041 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2/2 2024-09-15T00:33:32.041 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/2 2024-09-15T00:33:32.158 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 2.0 MB/s | 248 kB 00:00 2024-09-15T00:33:32.209 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-15T00:33:32.234 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-15T00:33:32.259 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-15T00:33:32.284 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): ceph-mgr-cephadm-19.1.1-324.gecab7be5.el 587 kB/s | 146 kB 00:00 2024-09-15T00:33:32.328 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T00:33:32.328 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:32.328 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:32.328 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T00:33:32.328 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:32.328 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:32.401 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-15T00:33:32.403 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:32.403 INFO:teuthology.orchestra.run.smithi059.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-15T00:33:32.477 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:32.488 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:32.595 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:32.595 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:32.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-volume 2024-09-15T00:33:32.927 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:33.101 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T00:33:33.163 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T00:33:33.208 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:33.350 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:33.351 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:33.351 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-09-15T00:33:33.351 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:33.351 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:33.351 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 285 k 2024-09-15T00:33:33.351 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:33.351 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:33.351 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:33.352 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-09-15T00:33:33.352 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:33.352 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 285 k 2024-09-15T00:33:33.352 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 1.3 M 2024-09-15T00:33:33.352 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:33.589 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-volume-19.1.1-324.gecab7be5.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-15T00:33:33.590 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:33.590 INFO:teuthology.orchestra.run.smithi002.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-15T00:33:33.590 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:33.599 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:33.599 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:33.631 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:33.631 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:33.716 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T00:33:33.810 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T00:33:33.864 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:33.880 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T00:33:33.906 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 6/6 2024-09-15T00:33:33.924 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:33:33.942 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:33:33.942 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:33:33.942 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T00:33:33.942 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:34.658 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 6/6 2024-09-15T00:33:34.658 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/6 2024-09-15T00:33:34.658 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T00:33:34.658 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T00:33:34.659 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T00:33:34.659 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T00:33:34.945 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T00:33:34.945 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:34.945 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:34.945 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:34.945 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T00:33:34.945 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T00:33:34.945 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T00:33:34.945 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T00:33:34.945 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T00:33:34.945 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:34.946 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:34.960 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:33:34.960 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:34.960 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:34.960 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:34.960 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:34.960 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:35.203 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2024-09-15T00:33:35.288 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install librados-devel 2024-09-15T00:33:35.792 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:35.885 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:35.932 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:35.932 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 896 k 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2024-09-15T00:33:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:35.934 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 976 k 2024-09-15T00:33:35.934 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 2.9 M 2024-09-15T00:33:35.934 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:36.026 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:36.026 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:36.026 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel x86_64 2:19.1.1-324.gecab7be5.el9 ceph 127 k 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 127 k 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 456 k 2024-09-15T00:33:36.027 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:36.244 INFO:teuthology.orchestra.run.smithi002.stdout:librados-devel-19.1.1-324.gecab7be5.el9.x86_64. 586 kB/s | 127 kB 00:00 2024-09-15T00:33:36.245 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:36.245 INFO:teuthology.orchestra.run.smithi002.stdout:Total 581 kB/s | 127 kB 00:00 2024-09-15T00:33:36.245 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:36.251 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:36.251 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:36.320 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:36.321 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:36.360 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 537 kB/s | 80 kB 00:00 2024-09-15T00:33:36.474 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:36.494 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): ceph-fuse-19.1.1-324.gecab7be5.el9.x86_6 3.1 MB/s | 896 kB 00:00 2024-09-15T00:33:36.494 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:36.495 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.7 MB/s | 976 kB 00:00 2024-09-15T00:33:36.507 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:36.531 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:36.532 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:36.579 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:36.619 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:36.619 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:36.811 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:36.950 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T00:33:37.021 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2/2 2024-09-15T00:33:37.050 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:37.293 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:37.293 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:37.293 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:37.294 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:37.294 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:37.294 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:37.571 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install libcephfs2 2024-09-15T00:33:37.707 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2/2 2024-09-15T00:33:37.707 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/2 2024-09-15T00:33:37.987 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T00:33:37.987 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:37.987 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:37.987 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T00:33:37.987 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:37.987 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:38.169 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:38.265 INFO:teuthology.orchestra.run.smithi002.stdout:Package libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:38.287 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-volume 2024-09-15T00:33:38.312 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:38.313 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-09-15T00:33:38.313 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:38.388 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install libcephfs-devel 2024-09-15T00:33:38.888 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:38.978 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:39.026 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:39.027 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:39.027 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-15T00:33:39.027 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:39.027 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:39.027 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 285 k 2024-09-15T00:33:39.027 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:39.027 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:39.027 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:39.027 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-15T00:33:39.027 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:39.028 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 285 k 2024-09-15T00:33:39.028 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 1.3 M 2024-09-15T00:33:39.028 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:39.118 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2024-09-15T00:33:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:39.118 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel x86_64 2:19.1.1-324.gecab7be5.el9 ceph 32 k 2024-09-15T00:33:39.118 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-09-15T00:33:39.118 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:39.119 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 32 k 2024-09-15T00:33:39.119 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 142 k 2024-09-15T00:33:39.119 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:39.397 INFO:teuthology.orchestra.run.smithi002.stdout:libcephfs-devel-19.1.1-324.gecab7be5.el9.x86_64 114 kB/s | 32 kB 00:00 2024-09-15T00:33:39.397 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:39.398 INFO:teuthology.orchestra.run.smithi002.stdout:Total 113 kB/s | 32 kB 00:00 2024-09-15T00:33:39.398 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:39.402 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:39.402 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:39.422 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:39.422 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:39.481 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-volume-19.1.1-324.gecab7be5.el9.noarch.rpm 630 kB/s | 285 kB 00:00 2024-09-15T00:33:39.481 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:39.481 INFO:teuthology.orchestra.run.smithi059.stdout:Total 628 kB/s | 285 kB 00:00 2024-09-15T00:33:39.482 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:39.491 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:39.491 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:39.517 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:39.524 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:39.525 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:39.614 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:39.760 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:39.824 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:33:39.842 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:33:39.842 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:33:39.842 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T00:33:39.842 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:40.013 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:40.250 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:40.251 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:40.251 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:40.251 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:40.251 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:40.251 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:40.464 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install librados2 2024-09-15T00:33:40.954 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-15T00:33:40.954 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:40.954 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:40.954 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-15T00:33:40.955 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:40.955 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:41.113 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:41.208 INFO:teuthology.orchestra.run.smithi002.stdout:Package librados2-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:41.247 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2024-09-15T00:33:41.254 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:41.255 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-09-15T00:33:41.255 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:41.338 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install librbd1 2024-09-15T00:33:41.859 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:45 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:41.937 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:41.999 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:41.999 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:41.999 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:19.1.1-324.gecab7be5.el9 ceph 127 k 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 127 k 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 456 k 2024-09-15T00:33:42.000 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:42.031 INFO:teuthology.orchestra.run.smithi002.stdout:Package librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:42.077 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:42.079 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-09-15T00:33:42.079 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:42.155 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-rados 2024-09-15T00:33:42.210 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-19.1.1-324.gecab7be5.el9.x86_64. 605 kB/s | 127 kB 00:00 2024-09-15T00:33:42.211 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:42.211 INFO:teuthology.orchestra.run.smithi059.stdout:Total 600 kB/s | 127 kB 00:00 2024-09-15T00:33:42.211 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:42.217 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:42.217 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:42.284 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:42.285 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:42.444 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:42.545 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:42.745 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:42.842 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:42.887 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:42.889 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-09-15T00:33:42.889 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:42.960 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:42.963 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-rgw 2024-09-15T00:33:43.219 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:43.219 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:43.220 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:43.220 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:43.220 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:43.220 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:43.462 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2024-09-15T00:33:43.555 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:43.649 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:43.695 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:43.696 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-09-15T00:33:43.696 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:43.774 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-cephfs 2024-09-15T00:33:44.061 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:48 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:44.155 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:44.201 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:44.202 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-15T00:33:44.203 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:44.274 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2024-09-15T00:33:44.364 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:45 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:44.458 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:44.503 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:44.504 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-09-15T00:33:44.504 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:44.587 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-rbd 2024-09-15T00:33:44.864 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:48 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:45.003 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:45.003 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:19.1.1-324.gecab7be5.el9 ceph 32 k 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 32 k 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 142 k 2024-09-15T00:33:45.004 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:45.155 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-19.1.1-324.gecab7be5.el9.x86_64 210 kB/s | 32 kB 00:00 2024-09-15T00:33:45.156 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:45.156 INFO:teuthology.orchestra.run.smithi059.stdout:Total 209 kB/s | 32 kB 00:00 2024-09-15T00:33:45.156 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:45.160 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:45.160 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:45.181 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:45.181 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:45.183 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:46 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:45.278 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:45.278 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:45.325 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:45.326 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-09-15T00:33:45.326 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:45.381 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:45.401 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install rbd-fuse 2024-09-15T00:33:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:45.985 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:45.985 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:45.985 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:45.985 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:45.985 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:45.985 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:45.991 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:46 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:46.132 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:46.132 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 85 k 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 85 k 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 231 k 2024-09-15T00:33:46.133 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:46.170 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2024-09-15T00:33:46.326 INFO:teuthology.orchestra.run.smithi002.stdout:rbd-fuse-19.1.1-324.gecab7be5.el9.x86_64.rpm 443 kB/s | 85 kB 00:00 2024-09-15T00:33:46.327 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:46.327 INFO:teuthology.orchestra.run.smithi002.stdout:Total 439 kB/s | 85 kB 00:00 2024-09-15T00:33:46.327 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:46.335 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:46.335 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:46.403 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:46.404 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:46.567 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:46.669 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:46.760 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:50 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:46.854 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:46.901 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:46.902 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-15T00:33:46.902 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:46.984 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2024-09-15T00:33:47.071 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:47.334 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:47.334 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:47.334 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:47.334 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:47.334 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:47.335 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:47.574 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:51 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:47.641 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install rbd-mirror 2024-09-15T00:33:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:47.716 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:47.717 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-15T00:33:47.717 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:47.789 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2024-09-15T00:33:48.241 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:49 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:48.382 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:48.382 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:48.382 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.1 M 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 3.1 M 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 13 M 2024-09-15T00:33:48.383 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:48.384 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:52 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:48.478 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:48.524 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:48.525 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-15T00:33:48.525 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:48.601 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2024-09-15T00:33:48.708 INFO:teuthology.orchestra.run.smithi002.stdout:rbd-mirror-19.1.1-324.gecab7be5.el9.x86_64.rpm 9.5 MB/s | 3.1 MB 00:00 2024-09-15T00:33:48.709 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:48.709 INFO:teuthology.orchestra.run.smithi002.stdout:Total 9.5 MB/s | 3.1 MB 00:00 2024-09-15T00:33:48.710 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:48.720 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:48.720 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:48.805 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:48.805 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:49.194 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:53 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:49.258 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:49.288 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:49.322 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:49.333 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:49.334 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-15T00:33:49.334 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:49.349 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:49.349 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:33:49.349 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T00:33:49.349 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T00:33:49.350 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T00:33:49.350 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:49.410 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2024-09-15T00:33:50.001 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:53 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:50.094 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:50.140 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:50.141 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-15T00:33:50.141 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:50.215 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2024-09-15T00:33:50.499 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:50.500 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:50.500 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:50.500 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:50.500 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:50.500 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:50.785 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install rbd-nbd 2024-09-15T00:33:50.811 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:54 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:50.921 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:50.972 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:50.973 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-15T00:33:50.973 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:51.052 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2024-09-15T00:33:51.393 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:52 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:51.534 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:51.535 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:51.535 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2024-09-15T00:33:51.535 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-15T00:33:51.536 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 170 k 2024-09-15T00:33:51.536 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-15T00:33:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-15T00:33:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-09-15T00:33:51.536 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 170 k 2024-09-15T00:33:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 494 k 2024-09-15T00:33:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-15T00:33:51.656 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:55 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:51.742 INFO:teuthology.orchestra.run.smithi002.stdout:rbd-nbd-19.1.1-324.gecab7be5.el9.x86_64.rpm 828 kB/s | 170 kB 00:00 2024-09-15T00:33:51.743 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:51.743 INFO:teuthology.orchestra.run.smithi002.stdout:Total 822 kB/s | 170 kB 00:00 2024-09-15T00:33:51.743 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-15T00:33:51.752 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-15T00:33:51.752 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-15T00:33:51.796 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:51.797 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-09-15T00:33:51.797 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:51.797 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 85 k 2024-09-15T00:33:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:51.797 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:51.797 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:51.797 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-15T00:33:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:51.798 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 85 k 2024-09-15T00:33:51.798 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 231 k 2024-09-15T00:33:51.798 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:51.834 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-15T00:33:51.834 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-15T00:33:51.992 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-19.1.1-324.gecab7be5.el9.x86_64.rpm 439 kB/s | 85 kB 00:00 2024-09-15T00:33:51.993 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-15T00:33:51.993 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:51.993 INFO:teuthology.orchestra.run.smithi059.stdout:Total 436 kB/s | 85 kB 00:00 2024-09-15T00:33:51.993 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:52.000 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:52.000 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:52.067 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:52.067 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:52.122 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:52.236 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:52.603 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:52.745 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:52.849 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:52.849 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:52.849 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-15T00:33:52.849 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:52.849 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:33:52.849 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:53.001 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:53.001 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:53.001 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:53.001 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:53.002 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:53.002 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:53.111 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install rbd-mirror 2024-09-15T00:33:53.264 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2024-09-15T00:33:53.707 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:54 ago on Sun 15 Sep 2024 12:31:59 AM UTC. 2024-09-15T00:33:53.803 INFO:teuthology.orchestra.run.smithi002.stdout:Package rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:53.849 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-15T00:33:53.850 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-09-15T00:33:53.851 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-15T00:33:53.865 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:57 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:53.934 DEBUG:teuthology.parallel:result is None 2024-09-15T00:33:54.007 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:54.008 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:54.008 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.1 M 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.1 M 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 13 M 2024-09-15T00:33:54.009 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:54.344 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-19.1.1-324.gecab7be5.el9.x86_64.rpm 9.3 MB/s | 3.1 MB 00:00 2024-09-15T00:33:54.345 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:54.345 INFO:teuthology.orchestra.run.smithi059.stdout:Total 9.2 MB/s | 3.1 MB 00:00 2024-09-15T00:33:54.345 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:54.355 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:54.355 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:54.441 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:54.441 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:54.953 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:54.980 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:54.980 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T00:33:54.980 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T00:33:54.981 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T00:33:54.981 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T00:33:54.981 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:56.042 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:56.042 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:56.042 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:56.043 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:56.043 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:56.043 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:56.316 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2024-09-15T00:33:56.912 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:00 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:57.052 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:57.053 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:57.053 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-09-15T00:33:57.053 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:57.053 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-15T00:33:57.053 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 170 k 2024-09-15T00:33:57.053 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:57.053 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-15T00:33:57.053 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-15T00:33:57.054 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-15T00:33:57.054 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:57.054 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 170 k 2024-09-15T00:33:57.054 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 494 k 2024-09-15T00:33:57.054 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-15T00:33:57.264 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-19.1.1-324.gecab7be5.el9.x86_64.rpm 815 kB/s | 170 kB 00:00 2024-09-15T00:33:57.264 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-15T00:33:57.264 INFO:teuthology.orchestra.run.smithi059.stdout:Total 808 kB/s | 170 kB 00:00 2024-09-15T00:33:57.264 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-15T00:33:57.273 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-15T00:33:57.273 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-15T00:33:57.341 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-15T00:33:57.341 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-15T00:33:57.504 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-15T00:33:57.615 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:58.031 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:58.293 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-15T00:33:58.293 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:58.293 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-15T00:33:58.293 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-15T00:33:58.293 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:33:58.293 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:58.576 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2024-09-15T00:33:59.175 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:03 ago on Sun 15 Sep 2024 12:31:56 AM UTC. 2024-09-15T00:33:59.271 INFO:teuthology.orchestra.run.smithi059.stdout:Package rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-15T00:33:59.321 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-15T00:33:59.322 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-15T00:33:59.323 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-15T00:33:59.399 DEBUG:teuthology.parallel:result is None 2024-09-15T00:33:59.400 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-15T00:33:59.548 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T00:33:59.574 INFO:teuthology.orchestra.run.smithi002.stdout:19.1.1-324.gecab7be5.el9 2024-09-15T00:33:59.575 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-324.gecab7be5.el9 2024-09-15T00:33:59.575 INFO:teuthology.task.install:The correct ceph version 19.1.1-324.gecab7be5 is installed. 2024-09-15T00:33:59.577 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-15T00:33:59.717 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T00:33:59.740 INFO:teuthology.orchestra.run.smithi059.stdout:19.1.1-324.gecab7be5.el9 2024-09-15T00:33:59.740 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-324.gecab7be5.el9 2024-09-15T00:33:59.740 INFO:teuthology.task.install:The correct ceph version 19.1.1-324.gecab7be5 is installed. 2024-09-15T00:33:59.742 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T00:33:59.742 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:33:59.742 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T00:33:59.774 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:33:59.774 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T00:33:59.807 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T00:33:59.807 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:33:59.807 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T00:33:59.843 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T00:33:59.913 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:33:59.913 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T00:33:59.940 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T00:34:00.007 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T00:34:00.007 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:00.007 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T00:34:00.036 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T00:34:00.104 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:00.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T00:34:00.131 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T00:34:00.198 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T00:34:00.198 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:00.198 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T00:34:00.226 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T00:34:00.294 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:00.294 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T00:34:00.320 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T00:34:00.385 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T00:34:00.459 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T00:34:00.460 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /var/log/ceph 2024-09-15T00:34:00.462 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /var/log/ceph 2024-09-15T00:34:00.488 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T00:34:00.488 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T00:34:00.531 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T00:34:00.557 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T00:34:00.557 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T00:34:00.599 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T00:34:00.636 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-15T00:34:00.636 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T00:34:00.636 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7904845', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 4271, 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'sleep_before_teardown': 0, 'subset': '97/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'targets': {'smithi002.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJKSh+HMyOt74GwrbujlVZhlgghNpatFlppFK6XAf1VJihRqZxdonpdkCxEi7ha9wNl2CQSxJFWt7DBlaGLl+4M=', 'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLJXXwvZdSj6pyFoClf2ZtIjKiyS/8t92Tv/1J/74+BOEC17P72ZXMB9q9sOaQ2LdvXVT8SeKmJQ75utNULR7Ok='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-14_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T00:34:00.637 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-15T00:34:00.667 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-15T00:34:00.682 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T00:34:00.725 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T00:34:00.750 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:00.751 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T00:34:00.807 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T00:34:00.807 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2024-09-15T00:34:00.864 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T00:34:00.864 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T00:34:00.864 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-15T00:34:00.864 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T00:34:00.864 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T00:34:00.864 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-15 00:33:49.607460915 +0000 2024-09-15T00:34:00.864 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-15 00:31:18.316268092 +0000 2024-09-15T00:34:00.864 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-15 00:31:18.316268092 +0000 2024-09-15T00:34:00.865 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-09-15 00:31:18.316268092 +0000 2024-09-15T00:34:00.865 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T00:34:00.930 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-15T00:34:00.930 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-15T00:34:00.930 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000103788 s, 4.9 MB/s 2024-09-15T00:34:00.931 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T00:34:00.989 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2024-09-15T00:34:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T00:34:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T00:34:01.046 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-15T00:34:01.046 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T00:34:01.046 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T00:34:01.046 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-15 00:33:49.608460893 +0000 2024-09-15T00:34:01.046 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-15 00:31:18.632260947 +0000 2024-09-15T00:34:01.046 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-15 00:31:18.632260947 +0000 2024-09-15T00:34:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-09-15 00:31:18.632260947 +0000 2024-09-15T00:34:01.047 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T00:34:01.112 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-15T00:34:01.112 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-15T00:34:01.112 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000106684 s, 4.8 MB/s 2024-09-15T00:34:01.113 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T00:34:01.170 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2024-09-15T00:34:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T00:34:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T00:34:01.227 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-15T00:34:01.227 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T00:34:01.227 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T00:34:01.227 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-15 00:33:49.608460893 +0000 2024-09-15T00:34:01.227 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-15 00:31:18.963253462 +0000 2024-09-15T00:34:01.227 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-15 00:31:18.963253462 +0000 2024-09-15T00:34:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-09-15 00:31:18.963253462 +0000 2024-09-15T00:34:01.227 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T00:34:01.291 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-15T00:34:01.292 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-15T00:34:01.292 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000104946 s, 4.9 MB/s 2024-09-15T00:34:01.293 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T00:34:01.350 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2024-09-15T00:34:01.406 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T00:34:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T00:34:01.407 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-15T00:34:01.407 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T00:34:01.407 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T00:34:01.407 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-15 00:33:49.608460893 +0000 2024-09-15T00:34:01.407 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-15 00:31:19.289246091 +0000 2024-09-15T00:34:01.407 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-15 00:31:19.289246091 +0000 2024-09-15T00:34:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-09-15 00:31:19.289246091 +0000 2024-09-15T00:34:01.408 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T00:34:01.472 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-15T00:34:01.472 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-15T00:34:01.472 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000105795 s, 4.8 MB/s 2024-09-15T00:34:01.473 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T00:34:01.530 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T00:34:01.530 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-15T00:34:01.530 INFO:tasks.ceph:Generating config... 2024-09-15T00:34:01.531 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client] log to stderr = False 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:01.532 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:01.533 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T00:34:01.533 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-15T00:34:01.533 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-15T00:34:01.533 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T00:34:01.533 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T00:34:01.533 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T00:34:01.533 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T00:34:01.533 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T00:34:01.533 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T00:34:01.533 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-15T00:34:01.533 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-15T00:34:01.533 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T00:34:01.533 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-15T00:34:01.533 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-15T00:34:01.533 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T00:34:01.534 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T00:34:01.534 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-15T00:34:01.534 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T00:34:01.534 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T00:34:01.534 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T00:34:01.534 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T00:34:01.534 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T00:34:01.534 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-15T00:34:01.534 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-15T00:34:01.622 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/cluster1.keyring 2024-09-15T00:34:01.625 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-15T00:34:01.669 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-15T00:34:01.735 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.2')] 2024-09-15T00:34:01.735 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.2', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-15T00:34:01.736 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:01.736 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T00:34:01.791 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.2 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-15T00:34:01.873 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-15T00:34:01.874 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: generated fsid 189ebd14-4d6c-4cd0-845c-b433081070db 2024-09-15T00:34:01.874 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = quincy 2024-09-15T00:34:01.874 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 0 2024-09-15T00:34:01.874 INFO:teuthology.orchestra.run.smithi002.stdout:fsid 189ebd14-4d6c-4cd0-845c-b433081070db 2024-09-15T00:34:01.874 INFO:teuthology.orchestra.run.smithi002.stdout:last_changed 2024-09-15T00:34:01.873170+0000 2024-09-15T00:34:01.874 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-09-15T00:34:01.873170+0000 2024-09-15T00:34:01.874 INFO:teuthology.orchestra.run.smithi002.stdout:min_mon_release 17 (quincy) 2024-09-15T00:34:01.874 INFO:teuthology.orchestra.run.smithi002.stdout:election_strategy: 1 2024-09-15T00:34:01.874 INFO:teuthology.orchestra.run.smithi002.stdout:0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-15T00:34:01.874 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-15T00:34:01.875 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T00:34:01.931 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 189ebd14-4d6c-4cd0-845c-b433081070db... 2024-09-15T00:34:01.932 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-15T00:34:01.973 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-15T00:34:02.017 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2024-09-15T00:34:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: chdir = "" 2024-09-15T00:34:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T00:34:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: auth supported = cephx 2024-09-15T00:34:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: filestore xattr use omap = true 2024-09-15T00:34:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: mon clock drift allowed = 1.000 2024-09-15T00:34:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: osd crush chooseleaf type = 0 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: auth debug = true 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on old message = true 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on bug = true 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: mon pg warn max object skew = 0 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default size = 2 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow primary affinity = true 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow pg remap = true 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on legacy crush tunables = false 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on crush straw calc version zero = false 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on no sortbitwise = false 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on osd down out interval zero = false 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on too few osds = false 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: mon_allow_pool_size_one = true 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: osd default data pool replay window = 5 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: mon allow pool delete = true 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: mon cluster log file level = debug 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: debug asserts on shutdown = true 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: mon health detail to clog = false 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: mon host = 172.21.15.2 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: mon client directed command retry = 5 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on pool no app = False 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: ms inject socket failures = 5000 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 189ebd14-4d6c-4cd0-845c-b433081070db 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout:[osd] 2024-09-15T00:34:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: osd journal size = 100 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub load threshold = 5.0 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub max interval = 600 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: osd mclock profile = high_recovery_ops 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: osd recover clone overlap = true 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: osd recovery max chunk = 1048576 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug shutdown = true 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug op order = true 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify stray on activate = true 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug trim objects = true 2024-09-15T00:34:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: osd open classes on start = true 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug pg log writeout = true 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: osd deep scrub update digest min age = 30 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: osd map max advance = 10 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: journal zero on create = true 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: filestore ondisk finisher threads = 3 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: filestore apply finisher threads = 3 2024-09-15T00:34:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: bdev debug aio = true 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug misdirected ops = true 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore block size = 96636764160 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore compression algorithm = lz4 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore compression mode = aggressive 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore fsck on mount = True 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: debug bluefs = 1/20 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: debug bluestore = 1/20 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: debug osd = 20 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: debug rocksdb = 4/10 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T00:34:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd full ratio = 0.9 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd nearfull ratio = 0.8 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: osd failsafe full ratio = 0.95 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: osd objectstore = bluestore 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr] 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: debug mgr = 20 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: mgr/telemetry/nag = false 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.024 INFO:teuthology.orchestra.run.smithi002.stdout:[mon] 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: debug paxos = 20 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: mon data avail warn = 5 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: mon mgr mkfs grace = 240 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd reporter subtree level = osd 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd prime pg temp = true 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T00:34:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: # 1m isn't quite enough 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: mon_down_mkfs_grace = 2m 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_filestore_osds = false 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout:[client] 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: rgw cache enabled = true 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable ops log = true 2024-09-15T00:34:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable usage log = true 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: debug rbd = 20 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: debug rbd_mirror = 15 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: log to stderr = False 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: rbd default features = 125 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: rbd mirror image policy type = none 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.a] 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.0] 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.1] 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:02.027 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.2] 2024-09-15T00:34:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:02.028 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.3] 2024-09-15T00:34:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:02.028 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-09-15T00:34:02.029 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-09-15T00:34:02.030 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-09-15T00:34:02.031 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-09-15T00:34:02.031 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-09-15T00:34:02.031 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-09-15T00:34:02.031 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-09-15T00:34:02.031 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T00:34:02.031 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = 172.21.15.2 2024-09-15T00:34:02.032 INFO:teuthology.orchestra.run.smithi059.stdout: mon client directed command retry = 5 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on pool no app = False 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: ms inject socket failures = 5000 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 189ebd14-4d6c-4cd0-845c-b433081070db 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-09-15T00:34:02.033 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-09-15T00:34:02.034 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.034 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-09-15T00:34:02.034 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-09-15T00:34:02.034 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-09-15T00:34:02.034 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-09-15T00:34:02.034 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.034 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-09-15T00:34:02.034 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-09-15T00:34:02.034 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.034 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore compression algorithm = lz4 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore compression mode = aggressive 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-09-15T00:34:02.035 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 1/20 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 1/20 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 4/10 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-09-15T00:34:02.036 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-09-15T00:34:02.037 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T00:34:02.038 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: debug rbd = 20 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: debug rbd_mirror = 15 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: log to stderr = False 2024-09-15T00:34:02.039 INFO:teuthology.orchestra.run.smithi059.stdout: rbd default features = 125 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout: rbd mirror image policy type = none 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.0] 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.1] 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.2] 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:02.040 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.3] 2024-09-15T00:34:02.041 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:02.041 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:02.042 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-15T00:34:02.042 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-15T00:34:02.137 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T00:34:02.137 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:02.137 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-15T00:34:02.153 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:02.153 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-15T00:34:02.208 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi002.front.sepia.ceph.com 2024-09-15T00:34:02.209 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:02.209 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-15T00:34:02.209 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-15T00:34:02.284 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:02.285 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-15T00:34:02.341 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T00:34:02.341 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:02.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-15T00:34:02.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-15T00:34:02.384 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:02.384 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-15T00:34:02.443 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T00:34:02.443 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T00:34:02.444 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-15T00:34:02.496 INFO:teuthology.orchestra.run.smithi002.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-15T00:34:02.498 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T00:34:02.499 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T00:34:02.499 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-15T00:34:02.541 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-15T00:34:02.554 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-15T00:34:02.609 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-15T00:34:02.624 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-15T00:34:02.669 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-15T00:34:02.683 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-15T00:34:02.726 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-15T00:34:02.740 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-15T00:34:02.822 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-15T00:34:02.837 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-15T00:34:02.920 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-15T00:34:02.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-15T00:34:03.017 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-15T00:34:03.030 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-15T00:34:03.112 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-15T00:34:03.126 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-15T00:34:03.208 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-15T00:34:03.221 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T00:34:03.221 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-15T00:34:03.222 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-15T00:34:03.253 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T00:34:03.253 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-15T00:34:03.253 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-15T00:34:03.254 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T00:34:03.323 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T00:34:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T00:34:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T00:34:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T00:34:03.323 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T00:34:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-09-15T00:34:03.323 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T00:34:03.323 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T00:34:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T00:34:03.324 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T00:34:03.356 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-09-15T00:34:03.358 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-09-15T00:34:03.358 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-15T00:34:03.429 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-15T00:34:03.500 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-15T00:34:03.568 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T00:34:03.568 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-15T00:34:03.568 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-15T00:34:03.568 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T00:34:03.635 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T00:34:03.635 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T00:34:03.635 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T00:34:03.635 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T00:34:03.635 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T00:34:03.635 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-09-15T00:34:03.636 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T00:34:03.636 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T00:34:03.636 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T00:34:03.636 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T00:34:03.667 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-09-15T00:34:03.669 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-09-15T00:34:03.669 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-15T00:34:03.740 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-15T00:34:03.810 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-15T00:34:03.877 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T00:34:03.878 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-15T00:34:03.878 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-15T00:34:03.878 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T00:34:03.945 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T00:34:03.945 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T00:34:03.945 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T00:34:03.945 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T00:34:03.945 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T00:34:03.945 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-09-15T00:34:03.945 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T00:34:03.945 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T00:34:03.945 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T00:34:03.945 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T00:34:03.978 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-09-15T00:34:03.980 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-09-15T00:34:03.980 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-15T00:34:04.054 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-15T00:34:04.124 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-15T00:34:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:04.217+0000 7f7a69a1e640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-15T00:34:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:04.218+0000 7f7a69a1e640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-15T00:34:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:04.218+0000 7f7a69a1e640 -1 bdev(0x5556fe407800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-15T00:34:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:04.218+0000 7f7a69a1e640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-15T00:34:07.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-15T00:34:07.925 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-15T00:34:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:08.015+0000 7f2596c14640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-15T00:34:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:08.015+0000 7f2596c14640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-15T00:34:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:08.016+0000 7f2596c14640 -1 bdev(0x55b96d39b800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-15T00:34:08.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:08.016+0000 7f2596c14640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-15T00:34:10.923 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-15T00:34:10.954 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-15T00:34:11.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:11.043+0000 7fe44260e640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-15T00:34:11.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:11.043+0000 7fe44260e640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-15T00:34:11.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:11.043+0000 7fe44260e640 -1 bdev(0x5634704d3800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-15T00:34:11.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-15T00:34:11.043+0000 7fe44260e640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-15T00:34:13.961 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-15T00:34:13.991 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T00:34:13.992 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:13.992 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-15T00:34:14.062 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:14.062 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-15T00:34:14.132 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:14.132 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-15T00:34:14.201 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:14.201 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-15T00:34:14.270 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:14.270 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-15T00:34:14.327 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:14.327 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-15T00:34:14.347 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:14.347 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-15T00:34:14.406 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:14.406 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-15T00:34:14.465 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:14.465 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-15T00:34:14.524 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:14.524 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-15T00:34:14.582 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:14.583 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-15T00:34:14.640 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:14.641 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-15T00:34:14.698 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:14.698 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-15T00:34:14.756 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T00:34:14.756 DEBUG:teuthology.orchestra.run.smithi002:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-15T00:34:14.784 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr.x] 2024-09-15T00:34:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB6K+ZmfzqLHRAAxBYt04yNZ7onNuDDuwaa1g== 2024-09-15T00:34:14.784 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.0] 2024-09-15T00:34:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB8K+ZmWSYIDRAAQQYNr5P4ZGxwQQewUKVxFQ== 2024-09-15T00:34:14.784 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.1] 2024-09-15T00:34:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQCAK+Zmg4n8ABAA3X5Y7q5tUwp2WNRIZzMq1g== 2024-09-15T00:34:14.784 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.2] 2024-09-15T00:34:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQCDK+ZmtbayAhAANXs6HKkez7rpwdI/REuchw== 2024-09-15T00:34:14.784 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB6K+ZmW144IBAAKX5c45Me0jkRwYxqgdQUvw== 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror] 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB6K+ZmOjlRJBAAew+O3ejmVR8BuxOogaw6lw== 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.0] 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB6K+Zm523lJxAAaUJY2v5BKLcdVrvpzVpC4w== 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.1] 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB6K+ZmCMNDKxAAP/8JE9iU3r1u2hGpVEiZIQ== 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.2] 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB6K+ZmJgIAMRAASZ/Ju+aU0TqkpnYfVhQU1w== 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.3] 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB6K+ZmklLNNhAAlk+9UlyzfZF0lBppktpLDQ== 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.4] 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB7K+ZmOZX/ABAAYvVFOKFdTpBQJuGIQdMe4g== 2024-09-15T00:34:14.785 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.5] 2024-09-15T00:34:14.786 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB7K+Zm9rGxBhAAh4AIU1TM0yv1370Ia8NkOA== 2024-09-15T00:34:14.786 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.6] 2024-09-15T00:34:14.786 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQB7K+Zm7Z1mDBAAyByFsccsFgLNHSCbeyz3PQ== 2024-09-15T00:34:14.786 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T00:34:14.886 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T00:34:14.939 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T00:34:14.991 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T00:34:15.043 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:15.097 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:15.153 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:15.207 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:15.262 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:15.317 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:15.372 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:15.427 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:15.481 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:15.536 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T00:34:15.536 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-15T00:34:15.562 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-15T00:34:15.659 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-15T00:34:15.684 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-15T00:34:15.740 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-15T00:34:15.740 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-15T00:34:15.740 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-15T00:34:15.782 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-15T00:34:15.782 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-15T00:34:15.783 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-15T00:34:15.783 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-15T00:34:15.785 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-15T00:34:15.785 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T00:34:15.785 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-15T00:34:15.961 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2024-09-15T00:34:15.972 INFO:tasks.ceph:check_enable_crimson: False 2024-09-15T00:34:15.972 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-15T00:34:15.972 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:15.972 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-15T00:34:15.998 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:15.998 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-15T00:34:16.064 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:16.064 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-15T00:34:16.133 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster cluster1 osd new 55e202c1-061d-4387-8c25-494b3be11d96 0 2024-09-15T00:34:16.160 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:16.158+0000 7f1c3d932040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T00:34:16.204 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:16.202+0000 7f1c3d932040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T00:34:16.343 INFO:teuthology.orchestra.run.smithi002.stdout:0 2024-09-15T00:34:16.355 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster cluster1 osd new 3f34fa1a-c511-44ce-8b21-08ca50cc3b2b 1 2024-09-15T00:34:16.357 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:16.354+0000 7f1c3d932040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T00:34:16.410 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:16.408+0000 7f1c3d932040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T00:34:16.535 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-09-15T00:34:16.546 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster cluster1 osd new 7f402869-0489-40b8-8fa2-dea899922e56 2 2024-09-15T00:34:16.595 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:16.593+0000 7f1c3d932040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T00:34:16.712 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:16.710+0000 7f1c3d932040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T00:34:16.728 INFO:teuthology.orchestra.run.smithi002.stdout:2 2024-09-15T00:34:16.739 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-15T00:34:16.739 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-15T00:34:16.741 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-15T00:34:16.741 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-15T00:34:16.741 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-15T00:34:16.743 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-15T00:34:16.743 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-15T00:34:16.744 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-15T00:34:16.745 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-15T00:34:16.746 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-15T00:34:16.985 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:16.985 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"fsid":"189ebd14-4d6c-4cd0-845c-b433081070db","created":"2024-09-15T00:34:15.869593+0000","modified":"2024-09-15T00:34:16.726844+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T00:34:16.726844+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"55e202c1-061d-4387-8c25-494b3be11d96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3f34fa1a-c511-44ce-8b21-08ca50cc3b2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f402869-0489-40b8-8fa2-dea899922e56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:34:17.000 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-15T00:34:17.000 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T00:34:17.113 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:17.111+0000 7f1c3d932040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T00:34:17.169 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:17.166+0000 7f1c3d932040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T00:34:17.303 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-15T00:34:17.325 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:17.323+0000 7f1c3d932040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T00:34:17.599 INFO:teuthology.misc.health.smithi002.stderr:2024-09-15T00:34:17.597+0000 7f13f48c8640 0 --2- 172.21.15.2:0/1671084184 >> v2:172.21.15.2:3300/0 conn(0x7f13f0073520 0x7f13f006fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T00:34:17.666 INFO:teuthology.misc.health.smithi002.stdout: 2024-09-15T00:34:17.667 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":5,"fsid":"189ebd14-4d6c-4cd0-845c-b433081070db","created":"2024-09-15T00:34:15.869593+0000","modified":"2024-09-15T00:34:16.726844+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T00:34:16.726844+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"55e202c1-061d-4387-8c25-494b3be11d96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3f34fa1a-c511-44ce-8b21-08ca50cc3b2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f402869-0489-40b8-8fa2-dea899922e56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:34:17.676 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-15T00:34:17.709 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:34:17.706+0000 7fbbd7040640 -1 Falling back to public interface 2024-09-15T00:34:17.712 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T00:34:17.710+0000 7fc086215640 -1 Falling back to public interface 2024-09-15T00:34:17.965 INFO:tasks.ceph.cluster1.osd.1.smithi002.stderr:2024-09-15T00:34:17.962+0000 7fc5e560f640 -1 Falling back to public interface 2024-09-15T00:34:18.331 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:18.329+0000 7f1c3d932040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T00:34:18.386 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:18.383+0000 7f1c3d932040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T00:34:18.768 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:18.766+0000 7f1c3d932040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T00:34:18.857 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:18.854+0000 7f1c3d932040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T00:34:18.968 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:18.966+0000 7f1c3d932040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T00:34:19.016 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:19.014+0000 7f1c3d932040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T00:34:19.137 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:19.132+0000 7f1c3d932040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T00:34:19.190 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:19.187+0000 7f1c3d932040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T00:34:19.236 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:19.234+0000 7f1c3d932040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T00:34:19.353 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:19.351+0000 7f1c3d932040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T00:34:19.405 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:19.403+0000 7f1c3d932040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T00:34:19.458 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:19.456+0000 7f1c3d932040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T00:34:19.529 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:19.526+0000 7f1c3d932040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T00:34:19.575 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:19.573+0000 7f1c3d932040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T00:34:19.621 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:19.619+0000 7f1c3d932040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T00:34:20.302 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:20.299+0000 7f1c3d932040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T00:34:21.309 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T00:34:21.307+0000 7fc086215640 -1 osd.2 0 log_to_monitors true 2024-09-15T00:34:21.361 INFO:tasks.ceph.cluster1.mgr.x.smithi002.stderr:2024-09-15T00:34:21.359+0000 7f1bdd796640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-15T00:34:21.519 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:34:21.516+0000 7fbbd7040640 -1 osd.0 0 log_to_monitors true 2024-09-15T00:34:22.009 INFO:tasks.ceph.cluster1.osd.1.smithi002.stderr:2024-09-15T00:34:22.007+0000 7fc5e560f640 -1 osd.1 0 log_to_monitors true 2024-09-15T00:34:23.434 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T00:34:23.431+0000 7fc081af5640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T00:34:23.446 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T00:34:23.445+0000 7fc079915640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T00:34:23.509 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:34:23.507+0000 7fbbd2fcf640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T00:34:23.520 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:34:23.519+0000 7fbbcadef640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T00:34:23.980 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-15T00:34:24.310 INFO:tasks.ceph.cluster1.osd.1.smithi002.stderr:2024-09-15T00:34:24.308+0000 7fc5e0ff9640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T00:34:24.323 INFO:tasks.ceph.cluster1.osd.1.smithi002.stderr:2024-09-15T00:34:24.322+0000 7fc5d8e19640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T00:34:24.337 INFO:teuthology.misc.health.smithi002.stdout: 2024-09-15T00:34:24.337 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":8,"fsid":"189ebd14-4d6c-4cd0-845c-b433081070db","created":"2024-09-15T00:34:15.869593+0000","modified":"2024-09-15T00:34:23.539144+0000","last_up_change":"2024-09-15T00:34:23.539144+0000","last_in_change":"2024-09-15T00:34:16.726844+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"55e202c1-061d-4387-8c25-494b3be11d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6801","nonce":3978604502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6803","nonce":3978604502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6807","nonce":3978604502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6805","nonce":3978604502}]},"public_addr":"172.21.15.2:6801/3978604502","cluster_addr":"172.21.15.2:6803/3978604502","heartbeat_back_addr":"172.21.15.2:6807/3978604502","heartbeat_front_addr":"172.21.15.2:6805/3978604502","state":["exists","up"]},{"osd":1,"uuid":"3f34fa1a-c511-44ce-8b21-08ca50cc3b2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f402869-0489-40b8-8fa2-dea899922e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6809","nonce":3675309926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6811","nonce":3675309926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6815","nonce":3675309926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6813","nonce":3675309926}]},"public_addr":"172.21.15.2:6809/3675309926","cluster_addr":"172.21.15.2:6811/3675309926","heartbeat_back_addr":"172.21.15.2:6815/3675309926","heartbeat_front_addr":"172.21.15.2:6813/3675309926","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:34:24.348 DEBUG:teuthology.misc:2 of 3 OSDs are up 2024-09-15T00:34:30.652 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-15T00:34:30.946 INFO:teuthology.misc.health.smithi002.stdout: 2024-09-15T00:34:30.946 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":11,"fsid":"189ebd14-4d6c-4cd0-845c-b433081070db","created":"2024-09-15T00:34:15.869593+0000","modified":"2024-09-15T00:34:26.554868+0000","last_up_change":"2024-09-15T00:34:24.549251+0000","last_in_change":"2024-09-15T00:34:16.726844+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T00:34:24.548876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55e202c1-061d-4387-8c25-494b3be11d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6801","nonce":3978604502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6803","nonce":3978604502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6807","nonce":3978604502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6805","nonce":3978604502}]},"public_addr":"172.21.15.2:6801/3978604502","cluster_addr":"172.21.15.2:6803/3978604502","heartbeat_back_addr":"172.21.15.2:6807/3978604502","heartbeat_front_addr":"172.21.15.2:6805/3978604502","state":["exists","up"]},{"osd":1,"uuid":"3f34fa1a-c511-44ce-8b21-08ca50cc3b2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6817","nonce":3851036120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6819","nonce":3851036120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6823","nonce":3851036120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6821","nonce":3851036120}]},"public_addr":"172.21.15.2:6817/3851036120","cluster_addr":"172.21.15.2:6819/3851036120","heartbeat_back_addr":"172.21.15.2:6823/3851036120","heartbeat_front_addr":"172.21.15.2:6821/3851036120","state":["exists","up"]},{"osd":2,"uuid":"7f402869-0489-40b8-8fa2-dea899922e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6809","nonce":3675309926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6811","nonce":3675309926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6815","nonce":3675309926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6813","nonce":3675309926}]},"public_addr":"172.21.15.2:6809/3675309926","cluster_addr":"172.21.15.2:6811/3675309926","heartbeat_back_addr":"172.21.15.2:6815/3675309926","heartbeat_front_addr":"172.21.15.2:6813/3675309926","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:22.477886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:23.048083+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:22.268062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:34:30.957 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-15T00:34:30.957 INFO:tasks.ceph:Creating RBD pool 2024-09-15T00:34:30.957 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-15T00:34:31.564 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'rbd' created 2024-09-15T00:34:31.579 DEBUG:teuthology.orchestra.run.smithi002:> rbd --cluster cluster1 pool init rbd 2024-09-15T00:34:34.583 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-15T00:34:34.583 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-15T00:34:34.583 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-15T00:34:34.584 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-15T00:34:34.584 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T00:34:34.932 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:34.943 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1639402624},{"type":"v1","addr":"172.21.15.2:6825","nonce":1639402624}]},"active_addr":"172.21.15.2:6825/1639402624","active_change":"2024-09-15T00:34:20.302618+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":31955573}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":380658386}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":596855026}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3563103471}]}]} 2024-09-15T00:34:34.945 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-15T00:34:34.945 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-15T00:34:34.945 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-15T00:34:35.231 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:35.231 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":15,"fsid":"189ebd14-4d6c-4cd0-845c-b433081070db","created":"2024-09-15T00:34:15.869593+0000","modified":"2024-09-15T00:34:34.570182+0000","last_up_change":"2024-09-15T00:34:24.549251+0000","last_in_change":"2024-09-15T00:34:16.726844+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T00:34:24.548876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T00:34:31.272465+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55e202c1-061d-4387-8c25-494b3be11d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6801","nonce":3978604502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6803","nonce":3978604502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6807","nonce":3978604502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6805","nonce":3978604502}]},"public_addr":"172.21.15.2:6801/3978604502","cluster_addr":"172.21.15.2:6803/3978604502","heartbeat_back_addr":"172.21.15.2:6807/3978604502","heartbeat_front_addr":"172.21.15.2:6805/3978604502","state":["exists","up"]},{"osd":1,"uuid":"3f34fa1a-c511-44ce-8b21-08ca50cc3b2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6817","nonce":3851036120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6819","nonce":3851036120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6823","nonce":3851036120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6821","nonce":3851036120}]},"public_addr":"172.21.15.2:6817/3851036120","cluster_addr":"172.21.15.2:6819/3851036120","heartbeat_back_addr":"172.21.15.2:6823/3851036120","heartbeat_front_addr":"172.21.15.2:6821/3851036120","state":["exists","up"]},{"osd":2,"uuid":"7f402869-0489-40b8-8fa2-dea899922e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6809","nonce":3675309926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6811","nonce":3675309926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6815","nonce":3675309926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6813","nonce":3675309926}]},"public_addr":"172.21.15.2:6809/3675309926","cluster_addr":"172.21.15.2:6811/3675309926","heartbeat_back_addr":"172.21.15.2:6815/3675309926","heartbeat_front_addr":"172.21.15.2:6813/3675309926","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:22.477886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:23.048083+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:22.268062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:34:35.243 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-15T00:34:35.244 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-15T00:34:35.532 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:35.532 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":15,"fsid":"189ebd14-4d6c-4cd0-845c-b433081070db","created":"2024-09-15T00:34:15.869593+0000","modified":"2024-09-15T00:34:34.570182+0000","last_up_change":"2024-09-15T00:34:24.549251+0000","last_in_change":"2024-09-15T00:34:16.726844+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T00:34:24.548876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T00:34:31.272465+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55e202c1-061d-4387-8c25-494b3be11d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6801","nonce":3978604502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6803","nonce":3978604502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6807","nonce":3978604502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6805","nonce":3978604502}]},"public_addr":"172.21.15.2:6801/3978604502","cluster_addr":"172.21.15.2:6803/3978604502","heartbeat_back_addr":"172.21.15.2:6807/3978604502","heartbeat_front_addr":"172.21.15.2:6805/3978604502","state":["exists","up"]},{"osd":1,"uuid":"3f34fa1a-c511-44ce-8b21-08ca50cc3b2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6817","nonce":3851036120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6819","nonce":3851036120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6823","nonce":3851036120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6821","nonce":3851036120}]},"public_addr":"172.21.15.2:6817/3851036120","cluster_addr":"172.21.15.2:6819/3851036120","heartbeat_back_addr":"172.21.15.2:6823/3851036120","heartbeat_front_addr":"172.21.15.2:6821/3851036120","state":["exists","up"]},{"osd":2,"uuid":"7f402869-0489-40b8-8fa2-dea899922e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6809","nonce":3675309926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6811","nonce":3675309926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6815","nonce":3675309926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6813","nonce":3675309926}]},"public_addr":"172.21.15.2:6809/3675309926","cluster_addr":"172.21.15.2:6811/3675309926","heartbeat_back_addr":"172.21.15.2:6815/3675309926","heartbeat_front_addr":"172.21.15.2:6813/3675309926","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:22.477886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:23.048083+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:22.268062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:34:35.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-15T00:34:35.545 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-15T00:34:35.545 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-15T00:34:35.685 INFO:teuthology.orchestra.run.smithi002.stdout:34359738373 2024-09-15T00:34:35.686 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-15T00:34:35.707 INFO:teuthology.orchestra.run.smithi002.stdout:38654705670 2024-09-15T00:34:35.708 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-15T00:34:35.709 INFO:teuthology.orchestra.run.smithi002.stdout:34359738373 2024-09-15T00:34:35.709 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-15T00:34:36.029 INFO:teuthology.orchestra.run.smithi002.stdout:34359738371 2024-09-15T00:34:36.042 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738371 for osd.2 2024-09-15T00:34:36.043 INFO:teuthology.orchestra.run.smithi002.stdout:38654705668 2024-09-15T00:34:36.056 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705670 got 38654705668 for osd.1 2024-09-15T00:34:36.088 INFO:teuthology.orchestra.run.smithi002.stdout:34359738372 2024-09-15T00:34:36.101 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2024-09-15T00:34:37.043 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-15T00:34:37.057 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-15T00:34:37.102 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-15T00:34:37.367 INFO:teuthology.orchestra.run.smithi002.stdout:34359738373 2024-09-15T00:34:37.381 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-09-15T00:34:37.381 DEBUG:teuthology.parallel:result is None 2024-09-15T00:34:37.400 INFO:teuthology.orchestra.run.smithi002.stdout:38654705670 2024-09-15T00:34:37.413 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705670 got 38654705670 for osd.1 2024-09-15T00:34:37.413 DEBUG:teuthology.parallel:result is None 2024-09-15T00:34:37.427 INFO:teuthology.orchestra.run.smithi002.stdout:34359738373 2024-09-15T00:34:37.439 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2024-09-15T00:34:37.440 DEBUG:teuthology.parallel:result is None 2024-09-15T00:34:37.440 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-15T00:34:37.440 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-15T00:34:37.766 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:37.766 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-15T00:34:37.779 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-15T00:34:36.309148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94088,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283021432,"statfs":{"total":289910292480,"available":289813946368,"internally_reserved":0,"allocated":516096,"data_stored":1296850,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.745245"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.578484+0000","last_change":"2024-09-15T00:34:34.578484+0000","last_active":"2024-09-15T00:34:34.578484+0000","last_peered":"2024-09-15T00:34:34.578484+0000","last_clean":"2024-09-15T00:34:34.578484+0000","last_became_active":"2024-09-15T00:34:32.575307+0000","last_became_peered":"2024-09-15T00:34:32.575307+0000","last_unstale":"2024-09-15T00:34:34.578484+0000","last_undegraded":"2024-09-15T00:34:34.578484+0000","last_fullsized":"2024-09-15T00:34:34.578484+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:04:44.235727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010984479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.578159+0000","last_change":"2024-09-15T00:34:34.578159+0000","last_active":"2024-09-15T00:34:34.578159+0000","last_peered":"2024-09-15T00:34:34.578159+0000","last_clean":"2024-09-15T00:34:34.578159+0000","last_became_active":"2024-09-15T00:34:32.576552+0000","last_became_peered":"2024-09-15T00:34:32.576552+0000","last_unstale":"2024-09-15T00:34:34.578159+0000","last_undegraded":"2024-09-15T00:34:34.578159+0000","last_fullsized":"2024-09-15T00:34:34.578159+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:22.053783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001076133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.577770+0000","last_change":"2024-09-15T00:34:34.577770+0000","last_active":"2024-09-15T00:34:34.577770+0000","last_peered":"2024-09-15T00:34:34.577770+0000","last_clean":"2024-09-15T00:34:34.577770+0000","last_became_active":"2024-09-15T00:34:32.574697+0000","last_became_peered":"2024-09-15T00:34:32.574697+0000","last_unstale":"2024-09-15T00:34:34.577770+0000","last_undegraded":"2024-09-15T00:34:34.577770+0000","last_fullsized":"2024-09-15T00:34:34.577770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:56:34.709213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073585800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.578167+0000","last_change":"2024-09-15T00:34:34.578167+0000","last_active":"2024-09-15T00:34:34.578167+0000","last_peered":"2024-09-15T00:34:34.578167+0000","last_clean":"2024-09-15T00:34:34.578167+0000","last_became_active":"2024-09-15T00:34:32.575384+0000","last_became_peered":"2024-09-15T00:34:32.575384+0000","last_unstale":"2024-09-15T00:34:34.578167+0000","last_undegraded":"2024-09-15T00:34:34.578167+0000","last_fullsized":"2024-09-15T00:34:34.578167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:06:32.098451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010254979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.574756+0000","last_change":"2024-09-15T00:34:34.574756+0000","last_active":"2024-09-15T00:34:34.574756+0000","last_peered":"2024-09-15T00:34:34.574756+0000","last_clean":"2024-09-15T00:34:34.574756+0000","last_became_active":"2024-09-15T00:34:32.575639+0000","last_became_peered":"2024-09-15T00:34:32.575639+0000","last_unstale":"2024-09-15T00:34:34.574756+0000","last_undegraded":"2024-09-15T00:34:34.574756+0000","last_fullsized":"2024-09-15T00:34:34.574756+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:44:05.451379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054626299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.912677+0000","last_change":"2024-09-15T00:34:34.912915+0000","last_active":"2024-09-15T00:34:34.912677+0000","last_peered":"2024-09-15T00:34:34.912677+0000","last_clean":"2024-09-15T00:34:34.912677+0000","last_became_active":"2024-09-15T00:34:32.573975+0000","last_became_peered":"2024-09-15T00:34:32.573975+0000","last_unstale":"2024-09-15T00:34:34.912677+0000","last_undegraded":"2024-09-15T00:34:34.912677+0000","last_fullsized":"2024-09-15T00:34:34.912677+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:42.456821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040528,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.912615+0000","last_change":"2024-09-15T00:34:34.912807+0000","last_active":"2024-09-15T00:34:34.912615+0000","last_peered":"2024-09-15T00:34:34.912615+0000","last_clean":"2024-09-15T00:34:34.912615+0000","last_became_active":"2024-09-15T00:34:32.573804+0000","last_became_peered":"2024-09-15T00:34:32.573804+0000","last_unstale":"2024-09-15T00:34:34.912615+0000","last_undegraded":"2024-09-15T00:34:34.912615+0000","last_fullsized":"2024-09-15T00:34:34.912615+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:29:40.374387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000336428,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.577734+0000","last_change":"2024-09-15T00:34:34.577734+0000","last_active":"2024-09-15T00:34:34.577734+0000","last_peered":"2024-09-15T00:34:34.577734+0000","last_clean":"2024-09-15T00:34:34.577734+0000","last_became_active":"2024-09-15T00:34:32.573483+0000","last_became_peered":"2024-09-15T00:34:32.573483+0000","last_unstale":"2024-09-15T00:34:34.577734+0000","last_undegraded":"2024-09-15T00:34:34.577734+0000","last_fullsized":"2024-09-15T00:34:34.577734+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:56:27.571672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067056699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.577272+0000","last_change":"2024-09-15T00:34:25.561004+0000","last_active":"2024-09-15T00:34:34.577272+0000","last_peered":"2024-09-15T00:34:34.577272+0000","last_clean":"2024-09-15T00:34:34.577272+0000","last_became_active":"2024-09-15T00:34:25.559814+0000","last_became_peered":"2024-09-15T00:34:25.559814+0000","last_unstale":"2024-09-15T00:34:34.577272+0000","last_undegraded":"2024-09-15T00:34:34.577272+0000","last_fullsized":"2024-09-15T00:34:34.577272+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_clean_scrub_stamp":"2024-09-15T00:34:24.549251+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:04:58.147756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31324,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340516,"statfs":{"total":96636764160,"available":96604688384,"internally_reserved":0,"allocated":131072,"data_stored":38692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T00:34:37.780 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-15T00:34:38.066 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:38.066 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-15T00:34:38.078 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-15T00:34:36.309148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94088,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283021432,"statfs":{"total":289910292480,"available":289813946368,"internally_reserved":0,"allocated":516096,"data_stored":1296850,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.745245"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.578484+0000","last_change":"2024-09-15T00:34:34.578484+0000","last_active":"2024-09-15T00:34:34.578484+0000","last_peered":"2024-09-15T00:34:34.578484+0000","last_clean":"2024-09-15T00:34:34.578484+0000","last_became_active":"2024-09-15T00:34:32.575307+0000","last_became_peered":"2024-09-15T00:34:32.575307+0000","last_unstale":"2024-09-15T00:34:34.578484+0000","last_undegraded":"2024-09-15T00:34:34.578484+0000","last_fullsized":"2024-09-15T00:34:34.578484+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:04:44.235727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010984479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.578159+0000","last_change":"2024-09-15T00:34:34.578159+0000","last_active":"2024-09-15T00:34:34.578159+0000","last_peered":"2024-09-15T00:34:34.578159+0000","last_clean":"2024-09-15T00:34:34.578159+0000","last_became_active":"2024-09-15T00:34:32.576552+0000","last_became_peered":"2024-09-15T00:34:32.576552+0000","last_unstale":"2024-09-15T00:34:34.578159+0000","last_undegraded":"2024-09-15T00:34:34.578159+0000","last_fullsized":"2024-09-15T00:34:34.578159+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:22.053783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001076133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.577770+0000","last_change":"2024-09-15T00:34:34.577770+0000","last_active":"2024-09-15T00:34:34.577770+0000","last_peered":"2024-09-15T00:34:34.577770+0000","last_clean":"2024-09-15T00:34:34.577770+0000","last_became_active":"2024-09-15T00:34:32.574697+0000","last_became_peered":"2024-09-15T00:34:32.574697+0000","last_unstale":"2024-09-15T00:34:34.577770+0000","last_undegraded":"2024-09-15T00:34:34.577770+0000","last_fullsized":"2024-09-15T00:34:34.577770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:56:34.709213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073585800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.578167+0000","last_change":"2024-09-15T00:34:34.578167+0000","last_active":"2024-09-15T00:34:34.578167+0000","last_peered":"2024-09-15T00:34:34.578167+0000","last_clean":"2024-09-15T00:34:34.578167+0000","last_became_active":"2024-09-15T00:34:32.575384+0000","last_became_peered":"2024-09-15T00:34:32.575384+0000","last_unstale":"2024-09-15T00:34:34.578167+0000","last_undegraded":"2024-09-15T00:34:34.578167+0000","last_fullsized":"2024-09-15T00:34:34.578167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:06:32.098451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010254979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.574756+0000","last_change":"2024-09-15T00:34:34.574756+0000","last_active":"2024-09-15T00:34:34.574756+0000","last_peered":"2024-09-15T00:34:34.574756+0000","last_clean":"2024-09-15T00:34:34.574756+0000","last_became_active":"2024-09-15T00:34:32.575639+0000","last_became_peered":"2024-09-15T00:34:32.575639+0000","last_unstale":"2024-09-15T00:34:34.574756+0000","last_undegraded":"2024-09-15T00:34:34.574756+0000","last_fullsized":"2024-09-15T00:34:34.574756+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:44:05.451379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054626299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.912677+0000","last_change":"2024-09-15T00:34:34.912915+0000","last_active":"2024-09-15T00:34:34.912677+0000","last_peered":"2024-09-15T00:34:34.912677+0000","last_clean":"2024-09-15T00:34:34.912677+0000","last_became_active":"2024-09-15T00:34:32.573975+0000","last_became_peered":"2024-09-15T00:34:32.573975+0000","last_unstale":"2024-09-15T00:34:34.912677+0000","last_undegraded":"2024-09-15T00:34:34.912677+0000","last_fullsized":"2024-09-15T00:34:34.912677+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:42.456821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040528,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.912615+0000","last_change":"2024-09-15T00:34:34.912807+0000","last_active":"2024-09-15T00:34:34.912615+0000","last_peered":"2024-09-15T00:34:34.912615+0000","last_clean":"2024-09-15T00:34:34.912615+0000","last_became_active":"2024-09-15T00:34:32.573804+0000","last_became_peered":"2024-09-15T00:34:32.573804+0000","last_unstale":"2024-09-15T00:34:34.912615+0000","last_undegraded":"2024-09-15T00:34:34.912615+0000","last_fullsized":"2024-09-15T00:34:34.912615+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:29:40.374387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000336428,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.577734+0000","last_change":"2024-09-15T00:34:34.577734+0000","last_active":"2024-09-15T00:34:34.577734+0000","last_peered":"2024-09-15T00:34:34.577734+0000","last_clean":"2024-09-15T00:34:34.577734+0000","last_became_active":"2024-09-15T00:34:32.573483+0000","last_became_peered":"2024-09-15T00:34:32.573483+0000","last_unstale":"2024-09-15T00:34:34.577734+0000","last_undegraded":"2024-09-15T00:34:34.577734+0000","last_fullsized":"2024-09-15T00:34:34.577734+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:56:27.571672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067056699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T00:34:34.577272+0000","last_change":"2024-09-15T00:34:25.561004+0000","last_active":"2024-09-15T00:34:34.577272+0000","last_peered":"2024-09-15T00:34:34.577272+0000","last_clean":"2024-09-15T00:34:34.577272+0000","last_became_active":"2024-09-15T00:34:25.559814+0000","last_became_peered":"2024-09-15T00:34:25.559814+0000","last_unstale":"2024-09-15T00:34:34.577272+0000","last_undegraded":"2024-09-15T00:34:34.577272+0000","last_fullsized":"2024-09-15T00:34:34.577272+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_clean_scrub_stamp":"2024-09-15T00:34:24.549251+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:04:58.147756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31324,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340516,"statfs":{"total":96636764160,"available":96604688384,"internally_reserved":0,"allocated":131072,"data_stored":38692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T00:34:38.078 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-15T00:34:38.079 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-15T00:34:38.079 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-15T00:34:38.079 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-15T00:34:38.403 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:38.403 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T00:34:38.416 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-15T00:34:38.416 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T00:34:38.425 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-15T00:34:38.425 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T00:34:38.426 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7904845', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 4271, 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'sleep_before_teardown': 0, 'subset': '97/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'targets': {'smithi002.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJKSh+HMyOt74GwrbujlVZhlgghNpatFlppFK6XAf1VJihRqZxdonpdkCxEi7ha9wNl2CQSxJFWt7DBlaGLl+4M=', 'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLJXXwvZdSj6pyFoClf2ZtIjKiyS/8t92Tv/1J/74+BOEC17P72ZXMB9q9sOaQ2LdvXVT8SeKmJQ75utNULR7Ok='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-14_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T00:34:38.426 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-15T00:34:38.429 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-15T00:34:38.448 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T00:34:38.485 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T00:34:38.520 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:38.520 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T00:34:38.580 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T00:34:38.580 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-09-15T00:34:38.637 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T00:34:38.638 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T00:34:38.638 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-15T00:34:38.638 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T00:34:38.638 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T00:34:38.638 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-15 00:33:55.234017004 +0000 2024-09-15T00:34:38.638 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-15 00:31:18.328735458 +0000 2024-09-15T00:34:38.638 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-15 00:31:18.328735458 +0000 2024-09-15T00:34:38.638 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-09-15 00:31:18.328735458 +0000 2024-09-15T00:34:38.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T00:34:38.704 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-15T00:34:38.704 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-15T00:34:38.704 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000215669 s, 2.4 MB/s 2024-09-15T00:34:38.705 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T00:34:38.764 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-09-15T00:34:38.822 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T00:34:38.822 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T00:34:38.822 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-15T00:34:38.822 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T00:34:38.822 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T00:34:38.822 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-15 00:33:55.235016991 +0000 2024-09-15T00:34:38.823 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-15 00:31:18.661731776 +0000 2024-09-15T00:34:38.823 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-15 00:31:18.661731776 +0000 2024-09-15T00:34:38.823 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-09-15 00:31:18.661731776 +0000 2024-09-15T00:34:38.823 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T00:34:38.889 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-15T00:34:38.889 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-15T00:34:38.889 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000202513 s, 2.5 MB/s 2024-09-15T00:34:38.890 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T00:34:38.947 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-09-15T00:34:39.003 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T00:34:39.004 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T00:34:39.004 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T00:34:39.004 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T00:34:39.004 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T00:34:39.004 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-15 00:33:55.235016991 +0000 2024-09-15T00:34:39.004 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-15 00:31:18.992728115 +0000 2024-09-15T00:34:39.004 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-15 00:31:18.992728115 +0000 2024-09-15T00:34:39.004 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-09-15 00:31:18.992728115 +0000 2024-09-15T00:34:39.004 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T00:34:39.069 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-15T00:34:39.069 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-15T00:34:39.069 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000214868 s, 2.4 MB/s 2024-09-15T00:34:39.070 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T00:34:39.128 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-09-15T00:34:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T00:34:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T00:34:39.184 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-15T00:34:39.184 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T00:34:39.184 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T00:34:39.184 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-15 00:33:55.236016980 +0000 2024-09-15T00:34:39.184 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-15 00:31:19.335724322 +0000 2024-09-15T00:34:39.184 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-15 00:31:19.335724322 +0000 2024-09-15T00:34:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-09-15 00:31:19.335724322 +0000 2024-09-15T00:34:39.184 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T00:34:39.249 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-15T00:34:39.249 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-15T00:34:39.249 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000189751 s, 2.7 MB/s 2024-09-15T00:34:39.251 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T00:34:39.309 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T00:34:39.309 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-15T00:34:39.309 INFO:tasks.ceph:Generating config... 2024-09-15T00:34:39.310 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client] log to stderr = False 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.311 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.312 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T00:34:39.312 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-15T00:34:39.312 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-15T00:34:39.312 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T00:34:39.312 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T00:34:39.312 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T00:34:39.312 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T00:34:39.312 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T00:34:39.312 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T00:34:39.312 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-15T00:34:39.312 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-15T00:34:39.312 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T00:34:39.312 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-15T00:34:39.312 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-15T00:34:39.313 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T00:34:39.313 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T00:34:39.313 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-15T00:34:39.313 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T00:34:39.313 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T00:34:39.313 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T00:34:39.313 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T00:34:39.313 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T00:34:39.313 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-15T00:34:39.313 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-15T00:34:39.391 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster2.keyring 2024-09-15T00:34:39.394 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-15T00:34:39.478 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-15T00:34:39.544 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.59')] 2024-09-15T00:34:39.545 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.59', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-15T00:34:39.545 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:39.545 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T00:34:39.600 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.59 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-15T00:34:39.682 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-15T00:34:39.682 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: generated fsid 201f1975-c83a-4da9-9090-536881f2f3b1 2024-09-15T00:34:39.683 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = quincy 2024-09-15T00:34:39.683 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 0 2024-09-15T00:34:39.683 INFO:teuthology.orchestra.run.smithi059.stdout:fsid 201f1975-c83a-4da9-9090-536881f2f3b1 2024-09-15T00:34:39.683 INFO:teuthology.orchestra.run.smithi059.stdout:last_changed 2024-09-15T00:34:39.681878+0000 2024-09-15T00:34:39.683 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-09-15T00:34:39.681878+0000 2024-09-15T00:34:39.683 INFO:teuthology.orchestra.run.smithi059.stdout:min_mon_release 17 (quincy) 2024-09-15T00:34:39.683 INFO:teuthology.orchestra.run.smithi059.stdout:election_strategy: 1 2024-09-15T00:34:39.683 INFO:teuthology.orchestra.run.smithi059.stdout:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2024-09-15T00:34:39.683 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-15T00:34:39.684 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T00:34:39.740 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 201f1975-c83a-4da9-9090-536881f2f3b1... 2024-09-15T00:34:39.741 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-15T00:34:39.744 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-15T00:34:39.794 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2024-09-15T00:34:39.794 INFO:teuthology.orchestra.run.smithi002.stdout: chdir = "" 2024-09-15T00:34:39.794 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T00:34:39.794 INFO:teuthology.orchestra.run.smithi002.stdout: auth supported = cephx 2024-09-15T00:34:39.794 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.794 INFO:teuthology.orchestra.run.smithi002.stdout: filestore xattr use omap = true 2024-09-15T00:34:39.794 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: mon clock drift allowed = 1.000 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: osd crush chooseleaf type = 0 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: auth debug = true 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on old message = true 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on bug = true 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: mon pg warn max object skew = 0 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T00:34:39.795 INFO:teuthology.orchestra.run.smithi002.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default size = 2 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow primary affinity = true 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow pg remap = true 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on legacy crush tunables = false 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on crush straw calc version zero = false 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on no sortbitwise = false 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on osd down out interval zero = false 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on too few osds = false 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: mon_allow_pool_size_one = true 2024-09-15T00:34:39.796 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: osd default data pool replay window = 5 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: mon allow pool delete = true 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: mon cluster log file level = debug 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: debug asserts on shutdown = true 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: mon health detail to clog = false 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: mon host = 172.21.15.59 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: mon client directed command retry = 5 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on pool no app = False 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: ms inject socket failures = 5000 2024-09-15T00:34:39.797 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 201f1975-c83a-4da9-9090-536881f2f3b1 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout:[osd] 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: osd journal size = 100 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub load threshold = 5.0 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub max interval = 600 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: osd mclock profile = high_recovery_ops 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: osd recover clone overlap = true 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: osd recovery max chunk = 1048576 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug shutdown = true 2024-09-15T00:34:39.798 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug op order = true 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify stray on activate = true 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug trim objects = true 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: osd open classes on start = true 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug pg log writeout = true 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: osd deep scrub update digest min age = 30 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: osd map max advance = 10 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: journal zero on create = true 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.799 INFO:teuthology.orchestra.run.smithi002.stdout: filestore ondisk finisher threads = 3 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: filestore apply finisher threads = 3 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: bdev debug aio = true 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug misdirected ops = true 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore block size = 96636764160 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore compression algorithm = lz4 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore compression mode = aggressive 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore fsck on mount = True 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: debug bluefs = 1/20 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: debug bluestore = 1/20 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: debug osd = 20 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: debug rocksdb = 4/10 2024-09-15T00:34:39.800 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd full ratio = 0.9 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd nearfull ratio = 0.8 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: osd failsafe full ratio = 0.95 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: osd objectstore = bluestore 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr] 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: debug mgr = 20 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2024-09-15T00:34:39.801 INFO:teuthology.orchestra.run.smithi002.stdout: mgr/telemetry/nag = false 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout:[mon] 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: debug paxos = 20 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: mon data avail warn = 5 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: mon mgr mkfs grace = 240 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd reporter subtree level = osd 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd prime pg temp = true 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: # 1m isn't quite enough 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: mon_down_mkfs_grace = 2m 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_filestore_osds = false 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout:[client] 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: rgw cache enabled = true 2024-09-15T00:34:39.803 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable ops log = true 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable usage log = true 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout: debug rbd = 20 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout: debug rbd_mirror = 15 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout: log to stderr = False 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout: rbd default features = 125 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout: rbd mirror image policy type = none 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.a] 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.0] 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.804 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.1] 2024-09-15T00:34:39.805 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.805 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.805 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.2] 2024-09-15T00:34:39.805 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.805 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.805 INFO:teuthology.orchestra.run.smithi002.stdout:[client.mirror.3] 2024-09-15T00:34:39.805 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.805 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.824 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-09-15T00:34:39.855 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-09-15T00:34:39.855 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T00:34:39.855 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-09-15T00:34:39.855 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.855 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-09-15T00:34:39.855 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T00:34:39.856 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-09-15T00:34:39.857 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = 172.21.15.59 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: mon client directed command retry = 5 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on pool no app = False 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: ms inject socket failures = 5000 2024-09-15T00:34:39.858 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 201f1975-c83a-4da9-9090-536881f2f3b1 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-09-15T00:34:39.859 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-09-15T00:34:39.860 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore compression algorithm = lz4 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore compression mode = aggressive 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 1/20 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 1/20 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 4/10 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-09-15T00:34:39.861 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-09-15T00:34:39.862 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T00:34:39.863 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T00:34:39.864 INFO:teuthology.orchestra.run.smithi059.stdout: debug rbd = 20 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout: debug rbd_mirror = 15 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout: log to stderr = False 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout: rbd default features = 125 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout: rbd mirror image policy type = none 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.0] 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.1] 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.2] 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.865 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.866 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.3] 2024-09-15T00:34:39.866 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-15T00:34:39.866 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-15T00:34:39.867 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-15T00:34:39.867 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-15T00:34:39.922 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T00:34:39.923 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:39.923 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-15T00:34:39.939 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:39.939 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-15T00:34:39.994 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi002.front.sepia.ceph.com 2024-09-15T00:34:39.994 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:39.994 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-15T00:34:39.994 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-15T00:34:40.036 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:40.036 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-15T00:34:40.094 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T00:34:40.094 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:40.094 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-15T00:34:40.094 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-15T00:34:40.130 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:40.130 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-15T00:34:40.186 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T00:34:40.186 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T00:34:40.187 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-15T00:34:40.244 INFO:teuthology.orchestra.run.smithi002.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-15T00:34:40.247 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T00:34:40.247 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T00:34:40.247 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-15T00:34:40.291 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-15T00:34:40.304 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-15T00:34:40.347 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-15T00:34:40.360 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-15T00:34:40.443 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-15T00:34:40.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-15T00:34:40.538 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-15T00:34:40.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-15T00:34:40.633 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-15T00:34:40.646 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-15T00:34:40.728 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-15T00:34:40.741 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-15T00:34:40.823 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-15T00:34:40.836 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-15T00:34:40.918 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-15T00:34:40.931 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-15T00:34:41.014 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-15T00:34:41.027 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T00:34:41.027 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-15T00:34:41.027 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-15T00:34:41.092 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T00:34:41.092 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-15T00:34:41.092 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T00:34:41.092 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T00:34:41.165 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T00:34:41.165 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T00:34:41.165 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T00:34:41.165 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T00:34:41.165 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T00:34:41.165 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-09-15T00:34:41.165 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T00:34:41.166 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T00:34:41.166 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T00:34:41.166 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T00:34:41.673 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-09-15T00:34:41.675 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-09-15T00:34:41.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-15T00:34:41.825 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-15T00:34:41.860 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-15T00:34:41.927 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T00:34:41.928 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-15T00:34:41.928 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T00:34:41.928 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T00:34:42.001 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T00:34:42.001 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T00:34:42.001 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T00:34:42.001 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T00:34:42.001 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T00:34:42.001 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-09-15T00:34:42.002 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T00:34:42.002 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T00:34:42.002 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T00:34:42.002 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T00:34:42.443 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-09-15T00:34:42.445 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-09-15T00:34:42.445 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-15T00:34:42.527 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-15T00:34:42.598 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-15T00:34:42.664 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T00:34:42.664 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-15T00:34:42.665 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T00:34:42.665 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T00:34:42.737 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T00:34:42.738 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T00:34:42.738 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T00:34:42.738 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T00:34:42.738 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T00:34:42.738 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-09-15T00:34:42.738 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T00:34:42.738 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T00:34:42.738 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T00:34:42.738 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T00:34:43.246 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-09-15T00:34:43.248 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-09-15T00:34:43.248 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-15T00:34:43.381 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-15T00:34:43.416 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-15T00:34:43.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:43.511+0000 7f0e6fc24640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-15T00:34:43.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:43.513+0000 7f0e6fc24640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-15T00:34:43.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:43.513+0000 7f0e6fc24640 -1 bdev(0x561206db5800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-15T00:34:43.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:43.513+0000 7f0e6fc24640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-15T00:34:46.437 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-15T00:34:46.468 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-15T00:34:46.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:46.558+0000 7f33c100f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-15T00:34:46.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:46.559+0000 7f33c100f640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-15T00:34:46.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:46.559+0000 7f33c100f640 -1 bdev(0x5623a6613800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-15T00:34:46.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:46.559+0000 7f33c100f640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-15T00:34:49.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-15T00:34:49.508 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-15T00:34:49.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:49.598+0000 7fb2c92f1640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-15T00:34:49.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:49.599+0000 7fb2c92f1640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-15T00:34:49.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:49.599+0000 7fb2c92f1640 -1 bdev(0x564f87fad800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-15T00:34:49.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-15T00:34:49.599+0000 7fb2c92f1640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-15T00:34:52.760 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-15T00:34:52.791 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T00:34:52.791 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:52.791 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-15T00:34:52.822 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:52.823 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-15T00:34:52.860 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:52.860 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-15T00:34:52.929 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:52.929 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-15T00:34:52.997 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-15T00:34:52.998 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-15T00:34:53.017 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:53.017 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-15T00:34:53.054 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:53.054 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-15T00:34:53.111 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:53.112 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-15T00:34:53.169 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:53.169 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-15T00:34:53.226 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:53.226 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-15T00:34:53.283 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:53.283 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-15T00:34:53.339 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:53.339 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-15T00:34:53.394 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:53.394 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-15T00:34:53.450 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T00:34:53.450 DEBUG:teuthology.orchestra.run.smithi059:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-15T00:34:53.515 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.x] 2024-09-15T00:34:53.515 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+ZmRWyPDhAAMDwnbkbFg9xBHh6SCy8n8g== 2024-09-15T00:34:53.515 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-09-15T00:34:53.515 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCjK+ZmCzmNHhAABIMQv/2gv9uthtrfFGxuaA== 2024-09-15T00:34:53.515 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCmK+Zmln1fIRAAW6gFqaVDLgsJ5xgz9rOP2g== 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCpK+ZmbtfAIxAAS+2y4CJSId38WGXIysC5eg== 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+ZmZNVWERAA4LjJAqj3Ns2wShp4bBbZjg== 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror] 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+ZmOoOnFBAAH1cpPD3ZqwQrH3ocIS2OOA== 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.0] 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+Zm8XppGhAArBwhdN+hdWW9vo/uiUnAwA== 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.1] 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+Zm1y0PIBAAK9Yzu38kMZF23tOJnFIlBA== 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.2] 2024-09-15T00:34:53.516 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+Zm/j27JRAAFm9QPZTim4sRSAsPCNgSFA== 2024-09-15T00:34:53.517 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.3] 2024-09-15T00:34:53.517 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+ZmwcZiKxAAESs5FzCgmQ2cfjylJPKHDA== 2024-09-15T00:34:53.517 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.4] 2024-09-15T00:34:53.517 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+Zm6qMOMRAA4N4LgxBpPgCrtZNkb+ODKA== 2024-09-15T00:34:53.517 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.5] 2024-09-15T00:34:53.517 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+Zmk5y5NhAAc9oWlJvCer0p+e5wPxXxOQ== 2024-09-15T00:34:53.517 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.6] 2024-09-15T00:34:53.517 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQChK+ZmGM7NABAAgj7vRjSrinTj7oLT2GRUzg== 2024-09-15T00:34:53.518 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T00:34:53.612 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T00:34:53.664 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T00:34:53.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T00:34:53.768 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:53.824 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:53.879 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:53.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:53.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:54.044 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:54.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:54.154 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:54.209 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T00:34:54.264 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T00:34:54.264 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-15T00:34:54.289 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-15T00:34:54.387 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-15T00:34:54.412 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-15T00:34:54.468 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-15T00:34:54.469 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-15T00:34:54.469 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-15T00:34:54.511 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-15T00:34:54.511 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-15T00:34:54.511 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-15T00:34:54.511 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-15T00:34:54.514 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-15T00:34:54.514 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T00:34:54.514 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-15T00:34:54.689 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-09-15T00:34:54.689 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:54.688+0000 7f5418325040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T00:34:54.700 INFO:tasks.ceph:check_enable_crimson: False 2024-09-15T00:34:54.700 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-15T00:34:54.700 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:54.700 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-15T00:34:54.726 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:54.726 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-15T00:34:54.732 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:54.731+0000 7f5418325040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T00:34:54.791 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:34:54.791 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-15T00:34:54.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster cluster2 osd new 8631fc02-9239-4ed9-a93e-68265cd02da9 0 2024-09-15T00:34:54.869 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:54.867+0000 7f5418325040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T00:34:54.919 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:54.918+0000 7f5418325040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T00:34:55.031 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:55.030+0000 7f5418325040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T00:34:55.077 INFO:teuthology.orchestra.run.smithi059.stdout:0 2024-09-15T00:34:55.088 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster cluster2 osd new 1d919c9a-dc94-45ee-b684-fe85740c1c3a 1 2024-09-15T00:34:55.128 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:55.127+0000 7f5418325040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T00:34:55.267 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-09-15T00:34:55.278 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster cluster2 osd new 3a30c237-ccd1-4b8e-9566-7e5f7c9f025f 2 2024-09-15T00:34:55.401 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:55.400+0000 7f5418325040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T00:34:55.447 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:55.445+0000 7f5418325040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T00:34:55.457 INFO:teuthology.orchestra.run.smithi059.stdout:2 2024-09-15T00:34:55.468 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-15T00:34:55.468 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-15T00:34:55.470 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-15T00:34:55.471 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-15T00:34:55.471 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-15T00:34:55.473 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-15T00:34:55.473 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-15T00:34:55.473 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-15T00:34:55.475 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-15T00:34:55.475 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-15T00:34:55.568 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:55.567+0000 7f5418325040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T00:34:55.651 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:34:55.652 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":5,"fsid":"201f1975-c83a-4da9-9090-536881f2f3b1","created":"2024-09-15T00:34:54.600851+0000","modified":"2024-09-15T00:34:55.456180+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T00:34:55.456180+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8631fc02-9239-4ed9-a93e-68265cd02da9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d919c9a-dc94-45ee-b684-fe85740c1c3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3a30c237-ccd1-4b8e-9566-7e5f7c9f025f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:34:55.662 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-15T00:34:55.663 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T00:34:55.965 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-15T00:34:56.130 INFO:teuthology.misc.health.smithi059.stdout: 2024-09-15T00:34:56.130 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":5,"fsid":"201f1975-c83a-4da9-9090-536881f2f3b1","created":"2024-09-15T00:34:54.600851+0000","modified":"2024-09-15T00:34:55.456180+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T00:34:55.456180+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8631fc02-9239-4ed9-a93e-68265cd02da9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d919c9a-dc94-45ee-b684-fe85740c1c3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3a30c237-ccd1-4b8e-9566-7e5f7c9f025f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:34:56.139 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-15T00:34:56.295 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:56.293+0000 7f5418325040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T00:34:56.341 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:56.339+0000 7f5418325040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T00:34:56.669 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:56.667+0000 7f5418325040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T00:34:56.674 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:34:56.672+0000 7f9976e0d640 -1 Falling back to public interface 2024-09-15T00:34:56.756 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:56.753+0000 7f5418325040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T00:34:56.867 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:56.865+0000 7f5418325040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T00:34:56.912 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:56.909+0000 7f5418325040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T00:34:56.912 INFO:tasks.ceph.cluster2.osd.0.smithi059.stderr:2024-09-15T00:34:56.910+0000 7f174cc9c640 -1 Falling back to public interface 2024-09-15T00:34:56.934 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:34:56.932+0000 7fc4ad233640 -1 Falling back to public interface 2024-09-15T00:34:57.027 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:57.025+0000 7f5418325040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T00:34:57.080 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:57.078+0000 7f5418325040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T00:34:57.124 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:57.121+0000 7f5418325040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T00:34:57.240 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:57.238+0000 7f5418325040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T00:34:57.290 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:57.288+0000 7f5418325040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T00:34:57.340 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:57.338+0000 7f5418325040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T00:34:57.395 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:57.392+0000 7f5418325040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T00:34:57.440 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:57.436+0000 7f5418325040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T00:34:57.482 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:57.480+0000 7f5418325040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T00:34:58.070 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:58.068+0000 7f5418325040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T00:34:59.080 INFO:tasks.ceph.cluster2.mgr.x.smithi002.stderr:2024-09-15T00:34:59.077+0000 7f53b82f3640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-15T00:35:00.217 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:35:00.216+0000 7f9976e0d640 -1 osd.1 0 log_to_monitors true 2024-09-15T00:35:00.661 INFO:tasks.ceph.cluster2.osd.0.smithi059.stderr:2024-09-15T00:35:00.659+0000 7f174cc9c640 -1 osd.0 0 log_to_monitors true 2024-09-15T00:35:01.207 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:35:01.205+0000 7fc4ad233640 -1 osd.2 0 log_to_monitors true 2024-09-15T00:35:02.443 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-15T00:35:02.856 INFO:teuthology.misc.health.smithi059.stdout: 2024-09-15T00:35:02.856 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":7,"fsid":"201f1975-c83a-4da9-9090-536881f2f3b1","created":"2024-09-15T00:34:54.600851+0000","modified":"2024-09-15T00:35:02.078394+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T00:34:55.456180+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8631fc02-9239-4ed9-a93e-68265cd02da9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d919c9a-dc94-45ee-b684-fe85740c1c3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3a30c237-ccd1-4b8e-9566-7e5f7c9f025f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:35:02.869 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-15T00:35:03.020 INFO:tasks.ceph.cluster2.osd.0.smithi059.stderr:2024-09-15T00:35:03.015+0000 7f1748c2b640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T00:35:03.030 INFO:tasks.ceph.cluster2.osd.0.smithi059.stderr:2024-09-15T00:35:03.029+0000 7f1740a4b640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T00:35:03.033 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:35:03.032+0000 7f997283a640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T00:35:03.045 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:35:03.044+0000 7f996a65a640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T00:35:03.794 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:35:03.792+0000 7fc4a91c2640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T00:35:03.804 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:35:03.803+0000 7fc4a0fe2640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T00:35:09.173 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-15T00:35:09.467 INFO:teuthology.misc.health.smithi059.stdout: 2024-09-15T00:35:09.467 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":12,"fsid":"201f1975-c83a-4da9-9090-536881f2f3b1","created":"2024-09-15T00:34:54.600851+0000","modified":"2024-09-15T00:35:07.094994+0000","last_up_change":"2024-09-15T00:35:04.086315+0000","last_in_change":"2024-09-15T00:34:55.456180+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T00:35:04.120349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8631fc02-9239-4ed9-a93e-68265cd02da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6809","nonce":4037059923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6811","nonce":4037059923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6815","nonce":4037059923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6813","nonce":4037059923}]},"public_addr":"172.21.15.59:6809/4037059923","cluster_addr":"172.21.15.59:6811/4037059923","heartbeat_back_addr":"172.21.15.59:6815/4037059923","heartbeat_front_addr":"172.21.15.59:6813/4037059923","state":["exists","up"]},{"osd":1,"uuid":"1d919c9a-dc94-45ee-b684-fe85740c1c3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6801","nonce":710469187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6803","nonce":710469187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6807","nonce":710469187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6805","nonce":710469187}]},"public_addr":"172.21.15.59:6801/710469187","cluster_addr":"172.21.15.59:6803/710469187","heartbeat_back_addr":"172.21.15.59:6807/710469187","heartbeat_front_addr":"172.21.15.59:6805/710469187","state":["exists","up"]},{"osd":2,"uuid":"3a30c237-ccd1-4b8e-9566-7e5f7c9f025f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6817","nonce":1104374936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6819","nonce":1104374936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6823","nonce":1104374936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6821","nonce":1104374936}]},"public_addr":"172.21.15.59:6817/1104374936","cluster_addr":"172.21.15.59:6819/1104374936","heartbeat_back_addr":"172.21.15.59:6823/1104374936","heartbeat_front_addr":"172.21.15.59:6821/1104374936","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:01.699919+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:01.184721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:02.247268+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:35:09.478 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-15T00:35:09.478 INFO:tasks.ceph:Creating RBD pool 2024-09-15T00:35:09.479 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-15T00:35:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'rbd' created 2024-09-15T00:35:10.124 DEBUG:teuthology.orchestra.run.smithi059:> rbd --cluster cluster2 pool init rbd 2024-09-15T00:35:13.136 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-15T00:35:13.136 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-15T00:35:13.137 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-15T00:35:13.137 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-15T00:35:13.137 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T00:35:13.485 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:35:13.496 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":107166991},{"type":"v1","addr":"172.21.15.2:6827","nonce":107166991}]},"active_addr":"172.21.15.2:6827/107166991","active_change":"2024-09-15T00:34:58.071534+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2025692241}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1530450211}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2123009317}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2862353115}]}]} 2024-09-15T00:35:13.498 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-15T00:35:13.498 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-15T00:35:13.498 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-15T00:35:13.786 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:35:13.786 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":16,"fsid":"201f1975-c83a-4da9-9090-536881f2f3b1","created":"2024-09-15T00:34:54.600851+0000","modified":"2024-09-15T00:35:13.123280+0000","last_up_change":"2024-09-15T00:35:04.086315+0000","last_in_change":"2024-09-15T00:34:55.456180+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T00:35:04.120349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T00:35:09.796195+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8631fc02-9239-4ed9-a93e-68265cd02da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6809","nonce":4037059923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6811","nonce":4037059923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6815","nonce":4037059923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6813","nonce":4037059923}]},"public_addr":"172.21.15.59:6809/4037059923","cluster_addr":"172.21.15.59:6811/4037059923","heartbeat_back_addr":"172.21.15.59:6815/4037059923","heartbeat_front_addr":"172.21.15.59:6813/4037059923","state":["exists","up"]},{"osd":1,"uuid":"1d919c9a-dc94-45ee-b684-fe85740c1c3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6801","nonce":710469187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6803","nonce":710469187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6807","nonce":710469187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6805","nonce":710469187}]},"public_addr":"172.21.15.59:6801/710469187","cluster_addr":"172.21.15.59:6803/710469187","heartbeat_back_addr":"172.21.15.59:6807/710469187","heartbeat_front_addr":"172.21.15.59:6805/710469187","state":["exists","up"]},{"osd":2,"uuid":"3a30c237-ccd1-4b8e-9566-7e5f7c9f025f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6817","nonce":1104374936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6819","nonce":1104374936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6823","nonce":1104374936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6821","nonce":1104374936}]},"public_addr":"172.21.15.59:6817/1104374936","cluster_addr":"172.21.15.59:6819/1104374936","heartbeat_back_addr":"172.21.15.59:6823/1104374936","heartbeat_front_addr":"172.21.15.59:6821/1104374936","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:01.699919+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:01.184721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:02.247268+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:35:13.798 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-15T00:35:13.799 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-15T00:35:14.086 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:35:14.086 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":16,"fsid":"201f1975-c83a-4da9-9090-536881f2f3b1","created":"2024-09-15T00:34:54.600851+0000","modified":"2024-09-15T00:35:13.123280+0000","last_up_change":"2024-09-15T00:35:04.086315+0000","last_in_change":"2024-09-15T00:34:55.456180+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T00:35:04.120349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T00:35:09.796195+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8631fc02-9239-4ed9-a93e-68265cd02da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6809","nonce":4037059923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6811","nonce":4037059923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6815","nonce":4037059923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6813","nonce":4037059923}]},"public_addr":"172.21.15.59:6809/4037059923","cluster_addr":"172.21.15.59:6811/4037059923","heartbeat_back_addr":"172.21.15.59:6815/4037059923","heartbeat_front_addr":"172.21.15.59:6813/4037059923","state":["exists","up"]},{"osd":1,"uuid":"1d919c9a-dc94-45ee-b684-fe85740c1c3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6801","nonce":710469187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6803","nonce":710469187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6807","nonce":710469187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6805","nonce":710469187}]},"public_addr":"172.21.15.59:6801/710469187","cluster_addr":"172.21.15.59:6803/710469187","heartbeat_back_addr":"172.21.15.59:6807/710469187","heartbeat_front_addr":"172.21.15.59:6805/710469187","state":["exists","up"]},{"osd":2,"uuid":"3a30c237-ccd1-4b8e-9566-7e5f7c9f025f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6817","nonce":1104374936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6819","nonce":1104374936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6823","nonce":1104374936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6821","nonce":1104374936}]},"public_addr":"172.21.15.59:6817/1104374936","cluster_addr":"172.21.15.59:6819/1104374936","heartbeat_back_addr":"172.21.15.59:6823/1104374936","heartbeat_front_addr":"172.21.15.59:6821/1104374936","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:01.699919+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:01.184721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:02.247268+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T00:35:14.098 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-15T00:35:14.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-15T00:35:14.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-15T00:35:14.239 INFO:teuthology.orchestra.run.smithi059.stdout:38654705669 2024-09-15T00:35:14.239 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-15T00:35:14.258 INFO:teuthology.orchestra.run.smithi059.stdout:34359738374 2024-09-15T00:35:14.259 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-15T00:35:14.260 INFO:teuthology.orchestra.run.smithi059.stdout:34359738374 2024-09-15T00:35:14.260 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-15T00:35:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:38654705668 2024-09-15T00:35:14.553 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.2 2024-09-15T00:35:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:34359738372 2024-09-15T00:35:14.601 INFO:teuthology.orchestra.run.smithi059.stdout:34359738372 2024-09-15T00:35:14.612 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.1 2024-09-15T00:35:14.614 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.0 2024-09-15T00:35:15.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-15T00:35:15.613 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-15T00:35:15.614 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-15T00:35:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:38654705668 2024-09-15T00:35:15.884 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.2 2024-09-15T00:35:15.920 INFO:teuthology.orchestra.run.smithi059.stdout:34359738372 2024-09-15T00:35:15.921 INFO:teuthology.orchestra.run.smithi059.stdout:34359738372 2024-09-15T00:35:15.933 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.1 2024-09-15T00:35:15.934 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.0 2024-09-15T00:35:16.885 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-15T00:35:16.934 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-15T00:35:16.934 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-15T00:35:17.201 INFO:teuthology.orchestra.run.smithi059.stdout:38654705670 2024-09-15T00:35:17.215 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705670 for osd.2 2024-09-15T00:35:17.215 DEBUG:teuthology.parallel:result is None 2024-09-15T00:35:17.238 INFO:teuthology.orchestra.run.smithi059.stdout:34359738374 2024-09-15T00:35:17.244 INFO:teuthology.orchestra.run.smithi059.stdout:34359738374 2024-09-15T00:35:17.251 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2024-09-15T00:35:17.251 DEBUG:teuthology.parallel:result is None 2024-09-15T00:35:17.257 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2024-09-15T00:35:17.257 DEBUG:teuthology.parallel:result is None 2024-09-15T00:35:17.258 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-15T00:35:17.258 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-15T00:35:17.585 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:35:17.585 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-15T00:35:17.600 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T00:35:16.078782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81620,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033900,"statfs":{"total":289910292480,"available":289826713600,"internally_reserved":0,"allocated":540672,"data_stored":1304053,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.970268"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.131214+0000","last_change":"2024-09-15T00:35:13.131214+0000","last_active":"2024-09-15T00:35:13.131214+0000","last_peered":"2024-09-15T00:35:13.131214+0000","last_clean":"2024-09-15T00:35:13.131214+0000","last_became_active":"2024-09-15T00:35:11.119667+0000","last_became_peered":"2024-09-15T00:35:11.119667+0000","last_unstale":"2024-09-15T00:35:13.131214+0000","last_undegraded":"2024-09-15T00:35:13.131214+0000","last_fullsized":"2024-09-15T00:35:13.131214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:56:15.752164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125682,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.131275+0000","last_change":"2024-09-15T00:35:13.131275+0000","last_active":"2024-09-15T00:35:13.131275+0000","last_peered":"2024-09-15T00:35:13.131275+0000","last_clean":"2024-09-15T00:35:13.131275+0000","last_became_active":"2024-09-15T00:35:11.119728+0000","last_became_peered":"2024-09-15T00:35:11.119728+0000","last_unstale":"2024-09-15T00:35:13.131275+0000","last_undegraded":"2024-09-15T00:35:13.131275+0000","last_fullsized":"2024-09-15T00:35:13.131275+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:38:32.666290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193798,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.131573+0000","last_change":"2024-09-15T00:35:13.131573+0000","last_active":"2024-09-15T00:35:13.131573+0000","last_peered":"2024-09-15T00:35:13.131573+0000","last_clean":"2024-09-15T00:35:13.131573+0000","last_became_active":"2024-09-15T00:35:11.120732+0000","last_became_peered":"2024-09-15T00:35:11.120732+0000","last_unstale":"2024-09-15T00:35:13.131573+0000","last_undegraded":"2024-09-15T00:35:13.131573+0000","last_fullsized":"2024-09-15T00:35:13.131573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:01:09.277084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001482562,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.130700+0000","last_change":"2024-09-15T00:35:13.130700+0000","last_active":"2024-09-15T00:35:13.130700+0000","last_peered":"2024-09-15T00:35:13.130700+0000","last_clean":"2024-09-15T00:35:13.130700+0000","last_became_active":"2024-09-15T00:35:11.120070+0000","last_became_peered":"2024-09-15T00:35:11.120070+0000","last_unstale":"2024-09-15T00:35:13.130700+0000","last_undegraded":"2024-09-15T00:35:13.130700+0000","last_fullsized":"2024-09-15T00:35:13.130700+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:26:49.101024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070255600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.129463+0000","last_change":"2024-09-15T00:35:13.128765+0000","last_active":"2024-09-15T00:35:13.129463+0000","last_peered":"2024-09-15T00:35:13.129463+0000","last_clean":"2024-09-15T00:35:13.129463+0000","last_became_active":"2024-09-15T00:35:11.120934+0000","last_became_peered":"2024-09-15T00:35:11.120934+0000","last_unstale":"2024-09-15T00:35:13.129463+0000","last_undegraded":"2024-09-15T00:35:13.129463+0000","last_fullsized":"2024-09-15T00:35:13.129463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:45:31.683827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011530049999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.345445+0000","last_change":"2024-09-15T00:35:13.345445+0000","last_active":"2024-09-15T00:35:13.345445+0000","last_peered":"2024-09-15T00:35:13.345445+0000","last_clean":"2024-09-15T00:35:13.345445+0000","last_became_active":"2024-09-15T00:35:11.123441+0000","last_became_peered":"2024-09-15T00:35:11.123441+0000","last_unstale":"2024-09-15T00:35:13.345445+0000","last_undegraded":"2024-09-15T00:35:13.345445+0000","last_fullsized":"2024-09-15T00:35:13.345445+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:34:52.254834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000350637,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.345476+0000","last_change":"2024-09-15T00:35:13.345476+0000","last_active":"2024-09-15T00:35:13.345476+0000","last_peered":"2024-09-15T00:35:13.345476+0000","last_clean":"2024-09-15T00:35:13.345476+0000","last_became_active":"2024-09-15T00:35:11.123456+0000","last_became_peered":"2024-09-15T00:35:11.123456+0000","last_unstale":"2024-09-15T00:35:13.345476+0000","last_undegraded":"2024-09-15T00:35:13.345476+0000","last_fullsized":"2024-09-15T00:35:13.345476+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:43.436651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038329899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.130729+0000","last_change":"2024-09-15T00:35:13.130729+0000","last_active":"2024-09-15T00:35:13.130729+0000","last_peered":"2024-09-15T00:35:13.130729+0000","last_clean":"2024-09-15T00:35:13.130729+0000","last_became_active":"2024-09-15T00:35:11.119438+0000","last_became_peered":"2024-09-15T00:35:11.119438+0000","last_unstale":"2024-09-15T00:35:13.130729+0000","last_undegraded":"2024-09-15T00:35:13.130729+0000","last_fullsized":"2024-09-15T00:35:13.130729+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:02:21.034948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071155299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.129846+0000","last_change":"2024-09-15T00:35:06.101340+0000","last_active":"2024-09-15T00:35:13.129846+0000","last_peered":"2024-09-15T00:35:13.129846+0000","last_clean":"2024-09-15T00:35:13.129846+0000","last_became_active":"2024-09-15T00:35:06.100362+0000","last_became_peered":"2024-09-15T00:35:06.100362+0000","last_unstale":"2024-09-15T00:35:13.129846+0000","last_undegraded":"2024-09-15T00:35:13.129846+0000","last_fullsized":"2024-09-15T00:35:13.129846+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_clean_scrub_stamp":"2024-09-15T00:35:05.089463+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:04.296940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":139264,"data_stored":41093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":631480,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":200704,"data_stored":631480,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T00:35:17.602 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-15T00:35:17.887 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:35:17.888 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-15T00:35:17.900 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T00:35:16.078782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81620,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033900,"statfs":{"total":289910292480,"available":289826713600,"internally_reserved":0,"allocated":540672,"data_stored":1304053,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.970268"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.131214+0000","last_change":"2024-09-15T00:35:13.131214+0000","last_active":"2024-09-15T00:35:13.131214+0000","last_peered":"2024-09-15T00:35:13.131214+0000","last_clean":"2024-09-15T00:35:13.131214+0000","last_became_active":"2024-09-15T00:35:11.119667+0000","last_became_peered":"2024-09-15T00:35:11.119667+0000","last_unstale":"2024-09-15T00:35:13.131214+0000","last_undegraded":"2024-09-15T00:35:13.131214+0000","last_fullsized":"2024-09-15T00:35:13.131214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:56:15.752164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125682,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.131275+0000","last_change":"2024-09-15T00:35:13.131275+0000","last_active":"2024-09-15T00:35:13.131275+0000","last_peered":"2024-09-15T00:35:13.131275+0000","last_clean":"2024-09-15T00:35:13.131275+0000","last_became_active":"2024-09-15T00:35:11.119728+0000","last_became_peered":"2024-09-15T00:35:11.119728+0000","last_unstale":"2024-09-15T00:35:13.131275+0000","last_undegraded":"2024-09-15T00:35:13.131275+0000","last_fullsized":"2024-09-15T00:35:13.131275+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:38:32.666290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193798,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.131573+0000","last_change":"2024-09-15T00:35:13.131573+0000","last_active":"2024-09-15T00:35:13.131573+0000","last_peered":"2024-09-15T00:35:13.131573+0000","last_clean":"2024-09-15T00:35:13.131573+0000","last_became_active":"2024-09-15T00:35:11.120732+0000","last_became_peered":"2024-09-15T00:35:11.120732+0000","last_unstale":"2024-09-15T00:35:13.131573+0000","last_undegraded":"2024-09-15T00:35:13.131573+0000","last_fullsized":"2024-09-15T00:35:13.131573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:01:09.277084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001482562,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.130700+0000","last_change":"2024-09-15T00:35:13.130700+0000","last_active":"2024-09-15T00:35:13.130700+0000","last_peered":"2024-09-15T00:35:13.130700+0000","last_clean":"2024-09-15T00:35:13.130700+0000","last_became_active":"2024-09-15T00:35:11.120070+0000","last_became_peered":"2024-09-15T00:35:11.120070+0000","last_unstale":"2024-09-15T00:35:13.130700+0000","last_undegraded":"2024-09-15T00:35:13.130700+0000","last_fullsized":"2024-09-15T00:35:13.130700+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:26:49.101024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070255600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.129463+0000","last_change":"2024-09-15T00:35:13.128765+0000","last_active":"2024-09-15T00:35:13.129463+0000","last_peered":"2024-09-15T00:35:13.129463+0000","last_clean":"2024-09-15T00:35:13.129463+0000","last_became_active":"2024-09-15T00:35:11.120934+0000","last_became_peered":"2024-09-15T00:35:11.120934+0000","last_unstale":"2024-09-15T00:35:13.129463+0000","last_undegraded":"2024-09-15T00:35:13.129463+0000","last_fullsized":"2024-09-15T00:35:13.129463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:45:31.683827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011530049999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.345445+0000","last_change":"2024-09-15T00:35:13.345445+0000","last_active":"2024-09-15T00:35:13.345445+0000","last_peered":"2024-09-15T00:35:13.345445+0000","last_clean":"2024-09-15T00:35:13.345445+0000","last_became_active":"2024-09-15T00:35:11.123441+0000","last_became_peered":"2024-09-15T00:35:11.123441+0000","last_unstale":"2024-09-15T00:35:13.345445+0000","last_undegraded":"2024-09-15T00:35:13.345445+0000","last_fullsized":"2024-09-15T00:35:13.345445+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:34:52.254834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000350637,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.345476+0000","last_change":"2024-09-15T00:35:13.345476+0000","last_active":"2024-09-15T00:35:13.345476+0000","last_peered":"2024-09-15T00:35:13.345476+0000","last_clean":"2024-09-15T00:35:13.345476+0000","last_became_active":"2024-09-15T00:35:11.123456+0000","last_became_peered":"2024-09-15T00:35:11.123456+0000","last_unstale":"2024-09-15T00:35:13.345476+0000","last_undegraded":"2024-09-15T00:35:13.345476+0000","last_fullsized":"2024-09-15T00:35:13.345476+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:43.436651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038329899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.130729+0000","last_change":"2024-09-15T00:35:13.130729+0000","last_active":"2024-09-15T00:35:13.130729+0000","last_peered":"2024-09-15T00:35:13.130729+0000","last_clean":"2024-09-15T00:35:13.130729+0000","last_became_active":"2024-09-15T00:35:11.119438+0000","last_became_peered":"2024-09-15T00:35:11.119438+0000","last_unstale":"2024-09-15T00:35:13.130729+0000","last_undegraded":"2024-09-15T00:35:13.130729+0000","last_fullsized":"2024-09-15T00:35:13.130729+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:02:21.034948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071155299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T00:35:13.129846+0000","last_change":"2024-09-15T00:35:06.101340+0000","last_active":"2024-09-15T00:35:13.129846+0000","last_peered":"2024-09-15T00:35:13.129846+0000","last_clean":"2024-09-15T00:35:13.129846+0000","last_became_active":"2024-09-15T00:35:06.100362+0000","last_became_peered":"2024-09-15T00:35:06.100362+0000","last_unstale":"2024-09-15T00:35:13.129846+0000","last_undegraded":"2024-09-15T00:35:13.129846+0000","last_fullsized":"2024-09-15T00:35:13.129846+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_clean_scrub_stamp":"2024-09-15T00:35:05.089463+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:04.296940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":139264,"data_stored":41093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":631480,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":200704,"data_stored":631480,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T00:35:17.901 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-15T00:35:17.901 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-15T00:35:17.901 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-15T00:35:17.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-15T00:35:18.227 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T00:35:18.227 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T00:35:18.240 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-15T00:35:18.240 INFO:teuthology.run_tasks:Running task exec... 2024-09-15T00:35:18.250 INFO:teuthology.task.exec:Executing custom commands... 2024-09-15T00:35:18.251 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T00:35:18.251 DEBUG:teuthology.orchestra.run.smithi059:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-15T00:35:18.619 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror] 2024-09-15T00:35:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQB6K+ZmOjlRJBAAew+O3ejmVR8BuxOogaw6lw== 2024-09-15T00:35:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-15T00:35:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "profile rbd" 2024-09-15T00:35:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.mirror 2024-09-15T00:35:18.633 DEBUG:teuthology.orchestra.run.smithi059:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-15T00:35:18.998 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.0] 2024-09-15T00:35:18.998 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQB6K+Zm523lJxAAaUJY2v5BKLcdVrvpzVpC4w== 2024-09-15T00:35:18.998 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "profile rbd-mirror" 2024-09-15T00:35:18.998 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "profile rbd" 2024-09-15T00:35:18.998 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.mirror.0 2024-09-15T00:35:19.011 DEBUG:teuthology.orchestra.run.smithi059:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-15T00:35:19.377 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.1] 2024-09-15T00:35:19.377 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQB6K+ZmCMNDKxAAP/8JE9iU3r1u2hGpVEiZIQ== 2024-09-15T00:35:19.377 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "profile rbd-mirror" 2024-09-15T00:35:19.377 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "profile rbd" 2024-09-15T00:35:19.377 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.mirror.1 2024-09-15T00:35:19.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-15T00:35:19.756 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.2] 2024-09-15T00:35:19.757 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQB6K+ZmJgIAMRAASZ/Ju+aU0TqkpnYfVhQU1w== 2024-09-15T00:35:19.757 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "profile rbd-mirror" 2024-09-15T00:35:19.757 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "profile rbd" 2024-09-15T00:35:19.757 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.mirror.2 2024-09-15T00:35:19.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-15T00:35:20.135 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.3] 2024-09-15T00:35:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQB6K+ZmklLNNhAAlk+9UlyzfZF0lBppktpLDQ== 2024-09-15T00:35:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "profile rbd-mirror" 2024-09-15T00:35:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "profile rbd" 2024-09-15T00:35:20.136 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.mirror.3 2024-09-15T00:35:20.150 DEBUG:teuthology.orchestra.run.smithi059:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-15T00:35:20.519 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror] 2024-09-15T00:35:20.519 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+ZmOoOnFBAAH1cpPD3ZqwQrH3ocIS2OOA== 2024-09-15T00:35:20.520 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-15T00:35:20.520 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "profile rbd" 2024-09-15T00:35:20.520 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.mirror 2024-09-15T00:35:20.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-15T00:35:20.905 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.0] 2024-09-15T00:35:20.905 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+Zm8XppGhAArBwhdN+hdWW9vo/uiUnAwA== 2024-09-15T00:35:20.905 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "profile rbd-mirror" 2024-09-15T00:35:20.905 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "profile rbd" 2024-09-15T00:35:20.906 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.mirror.0 2024-09-15T00:35:20.919 DEBUG:teuthology.orchestra.run.smithi059:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-15T00:35:21.292 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.1] 2024-09-15T00:35:21.292 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+Zm1y0PIBAAK9Yzu38kMZF23tOJnFIlBA== 2024-09-15T00:35:21.292 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "profile rbd-mirror" 2024-09-15T00:35:21.293 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "profile rbd" 2024-09-15T00:35:21.293 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.mirror.1 2024-09-15T00:35:21.306 DEBUG:teuthology.orchestra.run.smithi059:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-15T00:35:21.671 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.2] 2024-09-15T00:35:21.671 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+Zm/j27JRAAFm9QPZTim4sRSAsPCNgSFA== 2024-09-15T00:35:21.671 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "profile rbd-mirror" 2024-09-15T00:35:21.671 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "profile rbd" 2024-09-15T00:35:21.672 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.mirror.2 2024-09-15T00:35:21.685 DEBUG:teuthology.orchestra.run.smithi059:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-15T00:35:22.061 INFO:teuthology.orchestra.run.smithi059.stdout:[client.mirror.3] 2024-09-15T00:35:22.061 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCgK+ZmwcZiKxAAESs5FzCgmQ2cfjylJPKHDA== 2024-09-15T00:35:22.061 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "profile rbd-mirror" 2024-09-15T00:35:22.061 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "profile rbd" 2024-09-15T00:35:22.061 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.mirror.3 2024-09-15T00:35:22.075 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-15T00:35:22.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-15T00:35:22.086 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-15T00:35:22.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-15T00:35:22.089 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-15T00:35:22.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-15T00:35:22.165 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-15T00:35:22.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-15T00:35:22.168 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-15T00:35:22.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-15T00:35:22.248 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-15T00:35:22.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-15T00:35:22.255 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-15T00:35:22.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-15T00:35:22.322 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-15T00:35:22.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-15T00:35:22.324 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-15T00:35:22.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-15T00:35:22.381 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-15T00:35:22.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-15T00:35:22.383 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-15T00:35:22.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-15T00:35:22.453 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-15T00:35:22.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-15T00:35:22.456 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-15T00:35:22.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-15T00:35:22.539 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-15T00:35:22.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-15T00:35:22.542 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-15T00:35:22.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-15T00:35:22.615 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-15T00:35:22.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-15T00:35:22.617 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-15T00:35:22.691 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726360522 2024-09-15T00:35:22.691 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-15T00:35:22.691 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-15T00:35:22.732 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726360522 2024-09-15T00:35:22.732 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-15T00:35:22.732 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T00:35:22.781 INFO:tasks.workunit:Pulling workunits from ref ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-15T00:35:22.781 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T00:35:22.781 DEBUG:teuthology.orchestra.run.smithi059:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-15T00:35:22.782 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-15T00:35:22.782 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-15T00:35:22.782 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-15T00:35:22.782 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-15T00:35:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T00:35:22.799 INFO:teuthology.orchestra.run.smithi059.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-15T00:35:22.799 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-15T00:35:22.855 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-15T00:35:22.855 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-15T00:35:22.911 INFO:tasks.workunit:timeout=6h 2024-09-15T00:35:22.911 INFO:tasks.workunit:cleanup=True 2024-09-15T00:35:22.912 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-15T00:35:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-15T00:36:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Updating files: 95% (12328/12889) Updating files: 96% (12374/12889) Updating files: 97% (12503/12889) Updating files: 98% (12632/12889) Updating files: 99% (12761/12889) Updating files: 100% (12889/12889) Updating files: 100% (12889/12889), done. 2024-09-15T00:36:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Note: switching to 'ecab7be58963a5bd03910cd73eef66405d7fdd50'. 2024-09-15T00:36:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:36:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: git switch -c 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Or undo this operation with: 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:36:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: git switch - 2024-09-15T00:36:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:36:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T00:36:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:36:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:HEAD is now at ecab7be5896 Merge pull request #59767 from zdover23/wip-doc-2024-09-13-backport-59438-to-squid 2024-09-15T00:36:19.893 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-15T00:36:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-15T00:36:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-15T00:36:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-15T00:36:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-15T00:36:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-15T00:36:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-15T00:36:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-15T00:36:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-15T00:36:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-15T00:36:20.773 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-15T00:36:20.773 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-15T00:36:20.829 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-15T00:36:20.830 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-15T00:36:20.830 DEBUG:teuthology.orchestra.run.smithi059:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ecab7be58963a5bd03910cd73eef66405d7fdd50 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-15T00:36:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ IMAGE_COUNT=50 2024-09-15T00:36:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ export LOCKDEP=0 2024-09-15T00:36:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ LOCKDEP=0 2024-09-15T00:36:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-15T00:36:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ type xmlstarlet 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ XMLSTARLET=xmlstarlet 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CLUSTER1=cluster1 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CLUSTER2=cluster2 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ POOL=mirror 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ PARENT_POOL=mirror_parent 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ NS1=ns1 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ NS2=ns2 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ TEMPDIR= 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ID=mirror 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-15T00:36:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-15T00:36:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-15T00:36:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ MIRROR_POOL_MODE=image 2024-09-15T00:36:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' snapshot = snapshot ']' 2024-09-15T00:36:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ MIRROR_POOL_MODE=image 2024-09-15T00:36:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-15T00:36:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS='--id mirror' 2024-09-15T00:36:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-15T00:36:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-15T00:36:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-15T00:36:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-15T00:36:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_BIN=. 2024-09-15T00:36:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_SRC=. 2024-09-15T00:36:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-15T00:36:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 -gt 0 ']' 2024-09-15T00:36:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ setup 2024-09-15T00:36:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local c 2024-09-15T00:36:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-15T00:36:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ setup_tempdir 2024-09-15T00:36:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mktemp -d 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ TEMPDIR=/tmp/tmp.nS85ynkalX 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -z 1 ']' 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ setup_pools cluster1 cluster2 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local remote_cluster=cluster2 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mon_map_file 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mon_addr 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local admin_key_file 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local uuid 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T00:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-15T00:36:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:pool 'mirror' already exists 2024-09-15T00:36:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T00:36:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-15T00:36:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:pool 'mirror_parent' already exists 2024-09-15T00:36:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T00:36:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-15T00:36:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T00:36:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-15T00:36:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T00:36:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-15T00:36:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-15T00:36:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-15T00:36:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-15T00:36:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-15T00:36:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-15T00:36:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -z ']' 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -z ']' 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ peer_add cluster1 mirror cluster2 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local client_cluster=cluster2 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local remote_cluster=cluster2 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local uuid_var_name 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local error_code 2024-09-15T00:36:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local peer_uuid 2024-09-15T00:36:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 16 32 2024-09-15T00:36:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:36:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-15T00:36:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ peer_uuid=4196b0a9-f995-4e55-a9bb-2cbb2b437c4b 2024-09-15T00:36:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=0 2024-09-15T00:36:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:36:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 17 ']' 2024-09-15T00:36:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 -eq 0 2024-09-15T00:36:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:36:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-15T00:36:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:36:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror_parent 2024-09-15T00:36:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local client_cluster=cluster2 2024-09-15T00:36:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local remote_cluster=cluster2 2024-09-15T00:36:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local uuid_var_name 2024-09-15T00:36:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local error_code 2024-09-15T00:36:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local peer_uuid 2024-09-15T00:36:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 16 32 2024-09-15T00:36:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:36:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-15T00:36:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ peer_uuid=bbb26ded-8172-4e72-83f0-3aa17725f5c5 2024-09-15T00:36:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=0 2024-09-15T00:36:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:36:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 17 ']' 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 -eq 0 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ setup_pools cluster2 cluster1 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local remote_cluster=cluster1 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mon_map_file 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mon_addr 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local admin_key_file 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local uuid 2024-09-15T00:36:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T00:36:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-15T00:36:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:pool 'mirror' already exists 2024-09-15T00:36:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T00:36:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-15T00:36:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:pool 'mirror_parent' already exists 2024-09-15T00:36:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T00:36:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-15T00:36:31.783 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2814859834874366) > weight (0.25) 2024-09-15T00:36:31.783 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-15T00:36:31.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-15T00:36:31.784 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-15T00:36:31.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T00:36:31.783+0000 7fedb6a0b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 36605) UID: 1000 2024-09-15T00:36:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T00:36:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-15T00:36:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T00:36:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-15T00:36:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-15T00:36:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-15T00:36:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-15T00:36:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-15T00:36:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-15T00:36:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -z ']' 2024-09-15T00:36:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -z ']' 2024-09-15T00:36:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ peer_add cluster2 mirror cluster1 2024-09-15T00:36:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:36:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local client_cluster=cluster1 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local remote_cluster=cluster1 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local uuid_var_name 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local error_code 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local peer_uuid 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 16 32 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:36:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-15T00:36:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ peer_uuid=82ecb9a6-7d94-46cd-81a3-54f813198c11 2024-09-15T00:36:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=0 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 17 ']' 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 -eq 0 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror_parent 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local client_cluster=cluster1 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local remote_cluster=cluster1 2024-09-15T00:36:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local uuid_var_name 2024-09-15T00:36:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local error_code 2024-09-15T00:36:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local peer_uuid 2024-09-15T00:36:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 16 32 2024-09-15T00:36:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:36:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-15T00:36:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ peer_uuid=f31e99db-4429-46e4-b4dc-b16092b080e4 2024-09-15T00:36:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=0 2024-09-15T00:36:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:36:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 17 ']' 2024-09-15T00:36:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 -eq 0 2024-09-15T00:36:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:36:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:36:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirrors cluster1 2024-09-15T00:36:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:36:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 0 3 2024-09-15T00:36:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirror cluster1:0 2024-09-15T00:36:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1:0 2024-09-15T00:36:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance 2024-09-15T00:36:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:36:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local val=cluster1:0 2024-09-15T00:36:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster_var_name=cluster 2024-09-15T00:36:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance_var_name=instance 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cluster=cluster1 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ instance=0 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval cluster=cluster1 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval instance=0 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirror cluster1:1 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1:1 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:36:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local val=cluster1:1 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster_var_name=cluster 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance_var_name=instance 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cluster=cluster1 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ instance=1 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval cluster=cluster1 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval instance=1 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T00:36:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirror cluster1:2 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1:2 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local val=cluster1:2 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster_var_name=cluster 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance_var_name=instance 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cluster=cluster1 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ instance=2 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval cluster=cluster1 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval instance=2 2024-09-15T00:36:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirror cluster1:3 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1:3 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local val=cluster1:3 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster_var_name=cluster 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance_var_name=instance 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cluster=cluster1 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ instance=3 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval cluster=cluster1 2024-09-15T00:36:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval instance=3 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=3 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirrors cluster2 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 0 3 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirror cluster2:0 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2:0 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local val=cluster2:0 2024-09-15T00:36:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster_var_name=cluster 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance_var_name=instance 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cluster=cluster2 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ instance=0 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval cluster=cluster2 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster2 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval instance=0 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirror cluster2:1 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2:1 2024-09-15T00:36:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local val=cluster2:1 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster_var_name=cluster 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance_var_name=instance 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cluster=cluster2 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ instance=1 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval cluster=cluster2 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster2 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval instance=1 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T00:36:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirror cluster2:2 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2:2 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local val=cluster2:2 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster_var_name=cluster 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance_var_name=instance 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cluster=cluster2 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ instance=2 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval cluster=cluster2 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster2 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval instance=2 2024-09-15T00:36:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirror cluster2:3 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2:3 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local val=cluster2:3 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster_var_name=cluster 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance_var_name=instance 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cluster=cluster2 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ instance=3 2024-09-15T00:36:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval cluster=cluster2 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster2 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval instance=3 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=3 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ tee -a /tmp/tmp.nS85ynkalX/rbd-mirror.test.log 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ date '+%F %T' 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo 2024-09-15 00:36:37 TEST: add image and test replay after client crashes 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:2024-09-15 00:36:37 TEST: add image and test replay after client crashes 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=test 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:36:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror test 512M 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:36:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T00:36:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T00:36:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:36:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2024-09-15T00:36:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:36:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:36:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:36:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-15T00:36:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:36:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:36:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:36:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-15T00:36:37.784 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3819377022048769) > weight (0.25) 2024-09-15T00:36:37.784 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6655074839181833) > weight (0.25) 2024-09-15T00:36:37.784 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9604313031157441) > weight (0.25) 2024-09-15T00:36:37.784 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6535791864591088) > weight (0.25) 2024-09-15T00:36:37.784 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-15T00:36:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:36:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-15T00:36:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10adc5f9660d 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10adc5f9660d 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:37 2024 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:36:37 2024 2024-09-15T00:36:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:37 2024 2024-09-15T00:36:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:36:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:36:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:36:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:36:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:36:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:36:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:36:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:36:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:36:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:36:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:36:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:36:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:36:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:36:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:36:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:36:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:36:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:36:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:36:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:36:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:36:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:36:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:36:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:36:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:36:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:36:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:36:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:36:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:36:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:36:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:36:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:36:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:36:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:36:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:36:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:36:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:36:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:36:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:36:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:36:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:36:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:36:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:36:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:36:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:36:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:36:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:36:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:36:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:36:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:36:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:36:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:36:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:36:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:36:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:36:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:36:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:36:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:36:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-15T00:36:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:36:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:36:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:36:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:36:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:36:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:36:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:36:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:36:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:36:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:36:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:36:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:36:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-15T00:36:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:36:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:36:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:36:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-15T00:36:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:3 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:3 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=3 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=3 2024-09-15T00:36:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=3 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:3 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:3 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=3 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-15T00:36:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:36:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:36:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=3 2024-09-15T00:36:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=3 2024-09-15T00:36:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-15T00:36:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:36:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:36:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:36:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-15T00:36:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:36:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:36:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:36:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:36:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 1 2024-09-15T00:36:39.785 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-15T00:36:39.785 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:36:39.785 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-15T00:36:39.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-15T00:36:39.786 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-15T00:36:39.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-15T00:36:39.788 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-09-15T00:36:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:36:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:36:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:36:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:36:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:36:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:36:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:36:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:36:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:36:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:36:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:36:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:36:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:36:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:36:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:36:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:36:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:36:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:36:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:36:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:36:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:36:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:36:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:36:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:36:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:36:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:36:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:36:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:36:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:36:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:36:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:36:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:36:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:36:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:36:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:36:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:36:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:36:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:36:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:36:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:36:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:36:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:36:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:36:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:36:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:36:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:36:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:36:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:36:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-15T00:36:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:36:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:36:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:36:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:3 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:3 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=3 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=3 2024-09-15T00:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=3 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:3 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:3 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=3 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:36:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=3 2024-09-15T00:36:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=3 2024-09-15T00:36:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-15T00:36:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:36:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:36:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:36:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-15T00:36:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:36:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:36:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:36:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:36:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 2 2024-09-15T00:36:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:36:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:36:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:36:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:36:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:36:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:36:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:36:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:36:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:36:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:36:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:36:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:36:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:36:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:36:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:36:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:36:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:36:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:36:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:36:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:36:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:36:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:36:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:36:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:36:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:36:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:36:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:36:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:36:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:36:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:36:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:36:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:36:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:36:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:36:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:36:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:36:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:36:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name= 2024-09-15T00:36:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 10 2024-09-15T00:36:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:36:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror test 2024-09-15T00:36:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:36:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:36:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:36:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=14 2024-09-15T00:36:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:36:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-15T00:36:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 20515840 bytes 2024-09-15T00:36:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 66176000 bytes 2024-09-15T00:36:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 132849664 bytes 2024-09-15T00:36:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 177320960 bytes 2024-09-15T00:36:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 238905344 bytes 2024-09-15T00:36:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 296282112 bytes 2024-09-15T00:36:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 324049920 bytes 2024-09-15T00:36:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 375824384 bytes 2024-09-15T00:36:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 376469504 bytes 2024-09-15T00:36:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 401892352 bytes 2024-09-15T00:36:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 412839936 bytes 2024-09-15T00:36:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 438112256 bytes 2024-09-15T00:36:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 462880768 bytes 2024-09-15T00:36:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 476450816 bytes 2024-09-15T00:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 478451712 bytes 2024-09-15T00:36:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 502137856 bytes 2024-09-15T00:36:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 533853184 bytes 2024-09-15T00:36:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 552082432 bytes 2024-09-15T00:36:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 584757248 bytes 2024-09-15T00:36:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 600842240 bytes 2024-09-15T00:36:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 649969664 bytes 2024-09-15T00:36:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 694290432 bytes 2024-09-15T00:36:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 807300096 bytes 2024-09-15T00:36:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 887598080 bytes 2024-09-15T00:36:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 941952000 bytes 2024-09-15T00:36:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1003860992 bytes 2024-09-15T00:36:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1049869312 bytes 2024-09-15T00:36:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1050603520 bytes 2024-09-15T00:36:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1098065920 bytes 2024-09-15T00:36:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1113283584 bytes 2024-09-15T00:36:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1144608768 bytes 2024-09-15T00:36:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1164562432 bytes 2024-09-15T00:36:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1193943040 bytes 2024-09-15T00:36:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1229307904 bytes 2024-09-15T00:36:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1284197376 bytes 2024-09-15T00:36:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1387921408 bytes 2024-09-15T00:36:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1462380544 bytes 2024-09-15T00:36:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1462977536 bytes 2024-09-15T00:36:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1534135296 bytes 2024-09-15T00:36:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1580779520 bytes 2024-09-15T00:36:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1581405184 bytes 2024-09-15T00:36:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1615418368 bytes 2024-09-15T00:36:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1618340864 bytes 2024-09-15T00:36:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1635495936 bytes 2024-09-15T00:36:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1661566976 bytes 2024-09-15T00:36:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1731139584 bytes 2024-09-15T00:36:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1805259776 bytes 2024-09-15T00:36:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1885646848 bytes 2024-09-15T00:36:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1906908160 bytes 2024-09-15T00:36:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1951811584 bytes 2024-09-15T00:36:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1964505088 bytes 2024-09-15T00:36:48.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2021826560 bytes 2024-09-15T00:36:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2022446080 bytes 2024-09-15T00:36:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2075764736 bytes 2024-09-15T00:36:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2093375488 bytes 2024-09-15T00:36:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2143315968 bytes 2024-09-15T00:36:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2165331968 bytes 2024-09-15T00:36:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2237156352 bytes 2024-09-15T00:36:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2256555008 bytes 2024-09-15T00:36:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2298509312 bytes 2024-09-15T00:36:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2318189568 bytes 2024-09-15T00:36:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2369213440 bytes 2024-09-15T00:36:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2385830912 bytes 2024-09-15T00:36:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2413213696 bytes 2024-09-15T00:36:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2424465408 bytes 2024-09-15T00:36:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2457945088 bytes 2024-09-15T00:36:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2490238976 bytes 2024-09-15T00:36:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2524667904 bytes 2024-09-15T00:36:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2571927552 bytes 2024-09-15T00:36:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2572532736 bytes 2024-09-15T00:36:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2676820992 bytes 2024-09-15T00:36:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2701988864 bytes 2024-09-15T00:36:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2773351424 bytes 2024-09-15T00:36:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2786376704 bytes 2024-09-15T00:36:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2786993152 bytes 2024-09-15T00:36:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2828322816 bytes 2024-09-15T00:36:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 2854290432 bytes 2024-09-15T00:36:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 2854915072 bytes 2024-09-15T00:36:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 2867890176 bytes 2024-09-15T00:36:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 2904240128 bytes 2024-09-15T00:36:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 2909799424 bytes 2024-09-15T00:36:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 2946490368 bytes 2024-09-15T00:36:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3004891136 bytes 2024-09-15T00:36:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3061240832 bytes 2024-09-15T00:36:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3088796672 bytes 2024-09-15T00:36:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3129927680 bytes 2024-09-15T00:36:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3204630528 bytes 2024-09-15T00:36:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3209285632 bytes 2024-09-15T00:36:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3226615808 bytes 2024-09-15T00:36:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3248052224 bytes 2024-09-15T00:36:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3250290688 bytes 2024-09-15T00:36:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3312422912 bytes 2024-09-15T00:36:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3343703040 bytes 2024-09-15T00:36:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3357841408 bytes 2024-09-15T00:36:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 3421476864 bytes 2024-09-15T00:36:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 3446763520 bytes 2024-09-15T00:36:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 3515321344 bytes 2024-09-15T00:36:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 3540361216 bytes 2024-09-15T00:36:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 3545167872 bytes 2024-09-15T00:36:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 3627683840 bytes 2024-09-15T00:36:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 3674729472 bytes 2024-09-15T00:36:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 3807329280 bytes 2024-09-15T00:36:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 3838136320 bytes 2024-09-15T00:36:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 3859760128 bytes 2024-09-15T00:36:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 3921596416 bytes 2024-09-15T00:36:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 3940573184 bytes 2024-09-15T00:36:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4004148224 bytes 2024-09-15T00:36:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 4027328512 bytes 2024-09-15T00:36:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 4105661440 bytes 2024-09-15T00:36:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 4183458816 bytes 2024-09-15T00:36:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11100 IOs, 4212861952 bytes 2024-09-15T00:36:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11200 IOs, 4245452800 bytes 2024-09-15T00:36:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11300 IOs, 4270582784 bytes 2024-09-15T00:36:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11400 IOs, 4281838592 bytes 2024-09-15T00:36:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11500 IOs, 4328820736 bytes 2024-09-15T00:36:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11600 IOs, 4474771456 bytes 2024-09-15T00:36:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11700 IOs, 4534882304 bytes 2024-09-15T00:36:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11800 IOs, 4575748096 bytes 2024-09-15T00:36:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11900 IOs, 4576379904 bytes 2024-09-15T00:36:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12000 IOs, 4605353984 bytes 2024-09-15T00:36:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12100 IOs, 4638388224 bytes 2024-09-15T00:36:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12200 IOs, 4684643328 bytes 2024-09-15T00:36:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12300 IOs, 4708714496 bytes 2024-09-15T00:36:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12400 IOs, 4727842816 bytes 2024-09-15T00:36:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12500 IOs, 4798302208 bytes 2024-09-15T00:36:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12600 IOs, 4834465792 bytes 2024-09-15T00:36:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12700 IOs, 4870255616 bytes 2024-09-15T00:36:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12800 IOs, 4901871616 bytes 2024-09-15T00:36:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12900 IOs, 4935963648 bytes 2024-09-15T00:36:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13000 IOs, 4959899648 bytes 2024-09-15T00:36:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13100 IOs, 4992673792 bytes 2024-09-15T00:36:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13200 IOs, 5082716160 bytes 2024-09-15T00:36:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13300 IOs, 5128951808 bytes 2024-09-15T00:36:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13400 IOs, 5205404672 bytes 2024-09-15T00:36:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13500 IOs, 5271261184 bytes 2024-09-15T00:36:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13600 IOs, 5298061312 bytes 2024-09-15T00:36:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13700 IOs, 5305333760 bytes 2024-09-15T00:36:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13800 IOs, 5347965952 bytes 2024-09-15T00:36:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13900 IOs, 5350412288 bytes 2024-09-15T00:36:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:14000 IOs, 5351026688 bytes 2024-09-15T00:36:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:14100 IOs, 5370060800 bytes 2024-09-15T00:36:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:14200 IOs, 5390390272 bytes 2024-09-15T00:36:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:14300 IOs, 5509921792 bytes 2024-09-15T00:36:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:14400 IOs, 5528646656 bytes 2024-09-15T00:36:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:14500 IOs, 5564170240 bytes 2024-09-15T00:36:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:14600 IOs, 5601143808 bytes 2024-09-15T00:36:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:14700 IOs, 5707943936 bytes 2024-09-15T00:36:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:14800 IOs, 5747879936 bytes 2024-09-15T00:36:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:14900 IOs, 5798711296 bytes 2024-09-15T00:36:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:15000 IOs, 5830017024 bytes 2024-09-15T00:36:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:15100 IOs, 5863292928 bytes 2024-09-15T00:36:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:36:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:36:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:36:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:36:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-15T00:36:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:36:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:36:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:36:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:36:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:36:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:36:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:36:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:36:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:36:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:36:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:36:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:36:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:36:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:36:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:36:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:36:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:36:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:36:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:36:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: down+unknown 2024-09-15T00:36:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: status not found 2024-09-15T00:36:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15T00:36:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:36:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:36:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:36:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:36:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:36:54' 2024-09-15T00:36:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:36:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 1 2024-09-15T00:36:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:36:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 2 2024-09-15T00:37:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:37:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:37:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:37:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:37:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:37:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:37:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:37:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:37:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:37:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: down+unknown 2024-09-15T00:37:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: status not found 2024-09-15T00:37:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15T00:37:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:37:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:37:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:37:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:37:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:36:54' 2024-09-15T00:37:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:37:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 1 2024-09-15T00:37:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:37:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 4 2024-09-15T00:37:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:37:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:37:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:37:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:37:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:37:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:37:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:37:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:37:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:37:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: down+unknown 2024-09-15T00:37:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: status not found 2024-09-15T00:37:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15T00:37:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:37:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:37:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:37:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:37:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:36:54' 2024-09-15T00:37:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:37:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 1 2024-09-15T00:37:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:37:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 8 2024-09-15T00:37:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:37:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:37:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:37:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:37:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:37:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:37:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:37:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:37:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:37:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+replaying 2024-09-15T00:37:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360598,"remote_snapshot_timestamp":1726360598,"replay_state":"idle"} 2024-09-15T00:37:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: service: mirror.1 on smithi059 2024-09-15T00:37:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:37:09 2024-09-15T00:37:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:37:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:37:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:37:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:37:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:36:54' 2024-09-15T00:37:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:37:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: state: up+replaying 2024-09-15T00:37:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ description: .*' 2024-09-15T00:37:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360598,"remote_snapshot_timestamp":1726360598,"replay_state":"idle"} 2024-09-15T00:37:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:37:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:37:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^up+' 2024-09-15T00:37:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:37:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep 'service: *mirror..* on ' 2024-09-15T00:37:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: service: mirror.1 on smithi059 2024-09-15T00:37:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update 2024-09-15T00:37:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-15T00:37:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:37:09' 2024-09-15T00:37:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-15 00:37:09' 2024-09-15T00:37:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:37:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local 'prev_last_update=2024-09-15 00:37:09' 2024-09-15T00:37:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:37:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:37:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-15T00:37:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='OKOKOK142104293mirror.0smithi05919.1.1-324-gecab7be5falseOK42184294mirror.2smithi05919.1.1-324-gecab7be5falseOK42224295mirror.3smithi05919.1.1-324-gecab7be5falseOK42614264mirror.1smithi05919.1.1-324-gecab7be5trueOKtest23953c4f-e4d9-4660-9569-50c96f4cf0e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360598,"remote_snapshot_timestamp":1726360598,"replay_state":"idle"}42614264mirror.1smithi0592024-09-15 00:37:09cluster21108ad63-b7b4-4889-ae31-b1fd6de8431bup+stoppedlocal image is primary2024-09-15 00:36:54' 2024-09-15T00:37:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update state 2024-09-15T00:37:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-15T00:37:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:37:09' 2024-09-15T00:37:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-15T00:37:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=up+replaying 2024-09-15T00:37:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:37:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep up+replaying 2024-09-15T00:37:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:37:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:37:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:37:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap1 2024-09-15T00:37:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap1 2024-09-15T00:37:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:37:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap1 2024-09-15T00:37:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:37:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:37:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:37:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:37:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:37:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:37:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:37:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:37:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:37:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:37:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:37:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:37:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:37:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:37:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:37:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:37:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:37:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:37:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:37:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:37:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:37:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:37:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:37:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:37:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:37:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:37:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:37:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:37:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:37:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:37:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:37:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:37:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:37:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:37:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:37:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:37:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:37:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:37:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:37:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:37:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:37:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:37:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:37:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:37:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:37:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:37:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:37:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:37:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:37:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:37:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:37:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:37:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:37:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:37:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:37:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:37:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:37:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:37:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:37:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:37:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:37:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:37:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:37:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:37:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:37:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:37:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:37:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 5 2024-09-15T00:37:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:37:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:37:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:37:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:37:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:37:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:37:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=5 2024-09-15T00:37:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:37:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:37:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:37:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:37:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:37:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:37:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:37:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:37:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=3 2024-09-15T00:37:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 5 = 3 2024-09-15T00:37:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:37:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:37:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:37:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:37:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:37:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:37:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:37:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=3 2024-09-15T00:37:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 5 = 3 2024-09-15T00:37:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:37:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:37:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:37:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:37:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:37:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:37:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:37:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:37:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:37:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:37:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:37:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:37:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 5 = '' 2024-09-15T00:37:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:37:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:37:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:37:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:37:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:37:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:37:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:37:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=5 2024-09-15T00:37:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 5 = 5 2024-09-15T00:37:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:37:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-15T00:37:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap1 2024-09-15T00:37:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:37:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:37:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:37:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-15T00:37:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:37:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:37:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:37:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:37:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:37:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:37:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:37:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap1 2024-09-15T00:37:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap1 2024-09-15T00:37:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:37:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:37:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:37:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:37:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:37:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:37:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:37:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:37:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:37:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:37:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:37:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name=snap1-clean 2024-09-15T00:37:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-15T00:37:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:37:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap1-clean 2024-09-15T00:37:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-15T00:37:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:37:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror test 2024-09-15T00:37:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:37:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:37:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=14 2024-09-15T00:37:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:37:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-15T00:37:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 10737664 bytes 2024-09-15T00:37:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 98840576 bytes 2024-09-15T00:37:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 125110272 bytes 2024-09-15T00:37:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 173329408 bytes 2024-09-15T00:37:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 207236096 bytes 2024-09-15T00:37:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 268062720 bytes 2024-09-15T00:37:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 288120832 bytes 2024-09-15T00:37:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 302376960 bytes 2024-09-15T00:37:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 355830784 bytes 2024-09-15T00:37:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 396983296 bytes 2024-09-15T00:37:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 422428672 bytes 2024-09-15T00:37:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 558377984 bytes 2024-09-15T00:37:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 570721280 bytes 2024-09-15T00:37:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 604871680 bytes 2024-09-15T00:37:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 653733888 bytes 2024-09-15T00:37:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 663235584 bytes 2024-09-15T00:37:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 704566272 bytes 2024-09-15T00:37:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 722251776 bytes 2024-09-15T00:37:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 754149376 bytes 2024-09-15T00:37:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 791446528 bytes 2024-09-15T00:37:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 826214400 bytes 2024-09-15T00:37:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 887695360 bytes 2024-09-15T00:37:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 941094912 bytes 2024-09-15T00:37:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 970966016 bytes 2024-09-15T00:37:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1058928640 bytes 2024-09-15T00:37:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1148205056 bytes 2024-09-15T00:37:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1165857792 bytes 2024-09-15T00:37:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1200403456 bytes 2024-09-15T00:37:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1228540928 bytes 2024-09-15T00:37:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1261646848 bytes 2024-09-15T00:37:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1354687488 bytes 2024-09-15T00:37:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1414600704 bytes 2024-09-15T00:37:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1444421632 bytes 2024-09-15T00:37:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1499223040 bytes 2024-09-15T00:37:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1624133632 bytes 2024-09-15T00:37:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1715697664 bytes 2024-09-15T00:37:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1750710272 bytes 2024-09-15T00:37:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1831789568 bytes 2024-09-15T00:37:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1923851264 bytes 2024-09-15T00:37:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1982667776 bytes 2024-09-15T00:37:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 2017478656 bytes 2024-09-15T00:37:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 2089783296 bytes 2024-09-15T00:37:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2135223296 bytes 2024-09-15T00:37:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2158081024 bytes 2024-09-15T00:37:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2218332160 bytes 2024-09-15T00:37:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2222757888 bytes 2024-09-15T00:37:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2243137536 bytes 2024-09-15T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2313351168 bytes 2024-09-15T00:37:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2345433088 bytes 2024-09-15T00:37:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2346045440 bytes 2024-09-15T00:37:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2369576960 bytes 2024-09-15T00:37:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2400563200 bytes 2024-09-15T00:37:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2413064192 bytes 2024-09-15T00:37:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2445294592 bytes 2024-09-15T00:37:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2514669568 bytes 2024-09-15T00:37:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2581668864 bytes 2024-09-15T00:37:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2607047680 bytes 2024-09-15T00:37:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2679729152 bytes 2024-09-15T00:37:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2737591296 bytes 2024-09-15T00:37:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2778297344 bytes 2024-09-15T00:37:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2810706944 bytes 2024-09-15T00:37:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2865138688 bytes 2024-09-15T00:37:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2888010752 bytes 2024-09-15T00:37:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2946375680 bytes 2024-09-15T00:37:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2946989056 bytes 2024-09-15T00:37:43.785 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9675211487893227) > weight (0.25) 2024-09-15T00:37:43.786 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4759941048303391) > weight (0.25) 2024-09-15T00:37:43.786 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-15T00:37:43.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-15T00:37:43.786 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-15T00:37:43.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T00:37:43.786+0000 7f8918012640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 36805) UID: 1000 2024-09-15T00:37:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2980986880 bytes 2024-09-15T00:37:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3013768192 bytes 2024-09-15T00:37:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3048673280 bytes 2024-09-15T00:37:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3086171136 bytes 2024-09-15T00:37:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3118632960 bytes 2024-09-15T00:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3143057408 bytes 2024-09-15T00:37:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3205877760 bytes 2024-09-15T00:37:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3263917056 bytes 2024-09-15T00:37:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3321801728 bytes 2024-09-15T00:37:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3381326848 bytes 2024-09-15T00:37:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3427343360 bytes 2024-09-15T00:37:45.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3519536128 bytes 2024-09-15T00:37:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3617172480 bytes 2024-09-15T00:37:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3638090752 bytes 2024-09-15T00:37:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3666125824 bytes 2024-09-15T00:37:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3688718336 bytes 2024-09-15T00:37:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3709263872 bytes 2024-09-15T00:37:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3754145792 bytes 2024-09-15T00:37:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3805447168 bytes 2024-09-15T00:37:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3828636672 bytes 2024-09-15T00:37:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3883063296 bytes 2024-09-15T00:37:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3920560128 bytes 2024-09-15T00:37:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3957233664 bytes 2024-09-15T00:37:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3991094272 bytes 2024-09-15T00:37:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 4050695168 bytes 2024-09-15T00:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 4065375232 bytes 2024-09-15T00:37:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 4153906176 bytes 2024-09-15T00:37:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 4154510336 bytes 2024-09-15T00:37:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4194475008 bytes 2024-09-15T00:37:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4232502272 bytes 2024-09-15T00:37:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4291024896 bytes 2024-09-15T00:37:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4291643392 bytes 2024-09-15T00:37:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4296897536 bytes 2024-09-15T00:37:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4310151168 bytes 2024-09-15T00:37:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4341024768 bytes 2024-09-15T00:37:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4359995392 bytes 2024-09-15T00:37:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4370848768 bytes 2024-09-15T00:37:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4371451904 bytes 2024-09-15T00:37:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4426859520 bytes 2024-09-15T00:37:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4456528896 bytes 2024-09-15T00:37:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 4583209984 bytes 2024-09-15T00:37:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4627005440 bytes 2024-09-15T00:37:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 4655294464 bytes 2024-09-15T00:37:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 4663067648 bytes 2024-09-15T00:37:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 4690498560 bytes 2024-09-15T00:37:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11100 IOs, 4715509760 bytes 2024-09-15T00:37:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11200 IOs, 4826112000 bytes 2024-09-15T00:37:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11300 IOs, 4826723328 bytes 2024-09-15T00:37:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11400 IOs, 4888026112 bytes 2024-09-15T00:37:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11500 IOs, 4961183744 bytes 2024-09-15T00:37:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11600 IOs, 5053706240 bytes 2024-09-15T00:37:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11700 IOs, 5108174848 bytes 2024-09-15T00:37:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11800 IOs, 5196299264 bytes 2024-09-15T00:37:48.787 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-15T00:37:48.787 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:37:48.788 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-15T00:37:48.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-15T00:37:48.788 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-15T00:37:48.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-15T00:37:48.791 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-15T00:37:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11900 IOs, 5307405312 bytes 2024-09-15T00:37:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12000 IOs, 5374187520 bytes 2024-09-15T00:37:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12100 IOs, 5529239552 bytes 2024-09-15T00:37:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12200 IOs, 5598341120 bytes 2024-09-15T00:37:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12300 IOs, 5624349696 bytes 2024-09-15T00:37:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12400 IOs, 5638225920 bytes 2024-09-15T00:37:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12500 IOs, 5667575808 bytes 2024-09-15T00:37:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12600 IOs, 5721871360 bytes 2024-09-15T00:37:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12700 IOs, 5773241344 bytes 2024-09-15T00:37:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12800 IOs, 5788805120 bytes 2024-09-15T00:37:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12900 IOs, 5823940608 bytes 2024-09-15T00:37:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13000 IOs, 5846621184 bytes 2024-09-15T00:37:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13100 IOs, 5874989056 bytes 2024-09-15T00:37:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13200 IOs, 5894126592 bytes 2024-09-15T00:37:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13300 IOs, 5947153408 bytes 2024-09-15T00:37:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13400 IOs, 5999387648 bytes 2024-09-15T00:37:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13500 IOs, 6030210048 bytes 2024-09-15T00:37:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13600 IOs, 6061264896 bytes 2024-09-15T00:37:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13700 IOs, 6119315456 bytes 2024-09-15T00:37:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13800 IOs, 6199151616 bytes 2024-09-15T00:37:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13900 IOs, 6248827904 bytes 2024-09-15T00:37:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:37:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:37:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:37:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:37:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-15T00:37:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:37:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:37:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:37:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:37:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:37:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:37:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:37:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:37:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:37:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:37:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:37:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:37:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:37:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:37:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+replaying 2024-09-15T00:37:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: replaying, {"bytes_per_second":17882009.6,"bytes_per_snapshot":268230144.0,"last_snapshot_bytes":536460288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360650,"remote_snapshot_timestamp":1726360650,"replay_state":"idle"} 2024-09-15T00:37:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: service: mirror.1 on smithi059 2024-09-15T00:37:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:37:39 2024-09-15T00:37:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:37:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:37:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:37:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:37:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:37:22' 2024-09-15T00:37:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:37:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: state: up+replaying 2024-09-15T00:37:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ description: .*' 2024-09-15T00:37:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: description: replaying, {"bytes_per_second":17882009.6,"bytes_per_snapshot":268230144.0,"last_snapshot_bytes":536460288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360650,"remote_snapshot_timestamp":1726360650,"replay_state":"idle"} 2024-09-15T00:37:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:37:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:37:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^up+' 2024-09-15T00:37:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:37:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep 'service: *mirror..* on ' 2024-09-15T00:37:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: service: mirror.1 on smithi059 2024-09-15T00:37:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update 2024-09-15T00:37:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-15T00:37:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:37:39' 2024-09-15T00:37:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-15 00:37:39' 2024-09-15T00:37:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:37:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:37:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local 'prev_last_update=2024-09-15 00:37:39' 2024-09-15T00:37:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:37:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:37:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-15T00:37:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='OKOKOK142104293mirror.0smithi05919.1.1-324-gecab7be5falseOK42184294mirror.2smithi05919.1.1-324-gecab7be5falseOK42224295mirror.3smithi05919.1.1-324-gecab7be5falseOK42614264mirror.1smithi05919.1.1-324-gecab7be5trueOKtest23953c4f-e4d9-4660-9569-50c96f4cf0e9up+replayingreplaying, {"bytes_per_second":17882009.6,"bytes_per_snapshot":268230144.0,"last_snapshot_bytes":536460288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360650,"remote_snapshot_timestamp":1726360650,"replay_state":"idle"}42614264mirror.1smithi0592024-09-15 00:37:39cluster21108ad63-b7b4-4889-ae31-b1fd6de8431bup+stoppedlocal image is primary2024-09-15 00:37:22' 2024-09-15T00:37:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update state 2024-09-15T00:37:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-15T00:37:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:37:39' 2024-09-15T00:37:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-15T00:37:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=up+replaying 2024-09-15T00:37:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:37:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep up+replaying 2024-09-15T00:37:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:37:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:37:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:37:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap2 2024-09-15T00:37:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap2 2024-09-15T00:37:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:37:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:37:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:37:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap2 2024-09-15T00:37:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:38:21.789 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6525653784434481) > weight (0.25) 2024-09-15T00:38:21.790 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5296772697352577) > weight (0.25) 2024-09-15T00:38:21.790 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-15T00:38:21.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-15T00:38:21.791 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-15T00:38:21.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T00:38:21.790+0000 7f00b2185640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 36645) UID: 1000 2024-09-15T00:38:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:38:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:38:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:38:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:38:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:38:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:38:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:38:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:38:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:38:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:38:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:38:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:38:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:38:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:38:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:38:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:38:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:38:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:38:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:38:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:38:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:38:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:38:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:38:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:38:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:38:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:38:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:38:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:38:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:38:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:38:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:38:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:38:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:38:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:38:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:38:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:38:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:38:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:38:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:38:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:38:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:38:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:38:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:38:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:38:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:38:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:38:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:38:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:38:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:38:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:38:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:38:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:38:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:38:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:38:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:38:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:38:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:38:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:38:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:38:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:38:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:38:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 8 2024-09-15T00:38:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:38:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:38:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:38:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:38:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:38:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:38:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=8 2024-09-15T00:38:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:38:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:38:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:38:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:38:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:38:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:38:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:38:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:38:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:38:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:38:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:38:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:38:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:38:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 8 = '' 2024-09-15T00:38:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:38:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:38:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:38:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:38:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:38:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:38:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:38:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:38:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:38:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:38:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:38:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:38:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 8 = '' 2024-09-15T00:38:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:38:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:38:26.792 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-15T00:38:26.792 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:38:26.793 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-15T00:38:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-15T00:38:26.794 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-15T00:38:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-15T00:38:26.798 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-15T00:38:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:38:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:26.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:38:26.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:38:26.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:38:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:38:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:38:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:38:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:38:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:38:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:38:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 8 = '' 2024-09-15T00:38:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:38:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:38:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:38:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:38:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:38:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=8 2024-09-15T00:38:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 8 = 8 2024-09-15T00:38:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:38:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-15T00:38:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap2 2024-09-15T00:38:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:38:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:38:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:38:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-15T00:38:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:38:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 4 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:38:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:38:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:38:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:38:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:38:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:38:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snap1-clean ']' 2024-09-15T00:38:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-15T00:38:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap1-clean 2024-09-15T00:38:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:38:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:38:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:38:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:38:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:38:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:38:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:38:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:38:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:38:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:38:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:38:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap2 2024-09-15T00:38:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap2 2024-09-15T00:38:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:38:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:38:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:38:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:38:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:38:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:38:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:38:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:38:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:38:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name=snap2-clean 2024-09-15T00:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-15T00:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap2-clean 2024-09-15T00:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-15T00:38:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:38:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror test 2024-09-15T00:38:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:38:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:38:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=14 2024-09-15T00:38:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:38:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-15T00:38:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 38472704 bytes 2024-09-15T00:38:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 43389952 bytes 2024-09-15T00:38:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 114535424 bytes 2024-09-15T00:38:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 127731712 bytes 2024-09-15T00:38:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 177496064 bytes 2024-09-15T00:38:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 261191680 bytes 2024-09-15T00:38:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 293242880 bytes 2024-09-15T00:38:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 347843584 bytes 2024-09-15T00:38:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 403832832 bytes 2024-09-15T00:38:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 413824000 bytes 2024-09-15T00:38:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 485560320 bytes 2024-09-15T00:38:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 519365632 bytes 2024-09-15T00:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 615213056 bytes 2024-09-15T00:38:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 635544576 bytes 2024-09-15T00:38:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 687901696 bytes 2024-09-15T00:38:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 714910720 bytes 2024-09-15T00:38:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 779681792 bytes 2024-09-15T00:38:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 821842944 bytes 2024-09-15T00:38:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 862913536 bytes 2024-09-15T00:38:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 936928256 bytes 2024-09-15T00:38:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 966014976 bytes 2024-09-15T00:38:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1012135936 bytes 2024-09-15T00:38:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1047135232 bytes 2024-09-15T00:38:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1141464064 bytes 2024-09-15T00:38:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1163244544 bytes 2024-09-15T00:38:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1204228096 bytes 2024-09-15T00:38:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1258565632 bytes 2024-09-15T00:38:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1313035264 bytes 2024-09-15T00:38:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1326605312 bytes 2024-09-15T00:38:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1361709056 bytes 2024-09-15T00:38:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1408185344 bytes 2024-09-15T00:38:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1432753152 bytes 2024-09-15T00:38:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1474702336 bytes 2024-09-15T00:38:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1548696576 bytes 2024-09-15T00:38:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1581443072 bytes 2024-09-15T00:38:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1618677760 bytes 2024-09-15T00:38:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1628211200 bytes 2024-09-15T00:38:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1656894464 bytes 2024-09-15T00:38:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1688597504 bytes 2024-09-15T00:38:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1712062464 bytes 2024-09-15T00:38:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1762425856 bytes 2024-09-15T00:38:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1821373440 bytes 2024-09-15T00:38:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1830411264 bytes 2024-09-15T00:38:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1872666624 bytes 2024-09-15T00:38:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1873302528 bytes 2024-09-15T00:38:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1903560704 bytes 2024-09-15T00:38:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1914776576 bytes 2024-09-15T00:38:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1937210368 bytes 2024-09-15T00:38:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1993270272 bytes 2024-09-15T00:38:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2086888448 bytes 2024-09-15T00:38:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2102838272 bytes 2024-09-15T00:38:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2108153856 bytes 2024-09-15T00:38:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2157049856 bytes 2024-09-15T00:38:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2224154624 bytes 2024-09-15T00:38:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2224755712 bytes 2024-09-15T00:38:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2298545152 bytes 2024-09-15T00:38:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2367703040 bytes 2024-09-15T00:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2392255488 bytes 2024-09-15T00:38:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2462324736 bytes 2024-09-15T00:38:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2481794048 bytes 2024-09-15T00:38:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2515019776 bytes 2024-09-15T00:38:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2556414976 bytes 2024-09-15T00:38:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2613207040 bytes 2024-09-15T00:38:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2650143744 bytes 2024-09-15T00:38:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2682733568 bytes 2024-09-15T00:38:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2717605888 bytes 2024-09-15T00:38:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2756993024 bytes 2024-09-15T00:38:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2791507968 bytes 2024-09-15T00:38:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2815554560 bytes 2024-09-15T00:38:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2854279168 bytes 2024-09-15T00:38:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2907224064 bytes 2024-09-15T00:38:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2907855872 bytes 2024-09-15T00:38:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2959842304 bytes 2024-09-15T00:38:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2991194112 bytes 2024-09-15T00:38:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3035303936 bytes 2024-09-15T00:38:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3066798080 bytes 2024-09-15T00:38:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3137263616 bytes 2024-09-15T00:38:41.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3137897472 bytes 2024-09-15T00:38:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3192964096 bytes 2024-09-15T00:38:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3248761856 bytes 2024-09-15T00:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3325815808 bytes 2024-09-15T00:38:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3396149248 bytes 2024-09-15T00:38:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3452483584 bytes 2024-09-15T00:38:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3469105152 bytes 2024-09-15T00:38:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3506674688 bytes 2024-09-15T00:38:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3571311616 bytes 2024-09-15T00:38:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3626170368 bytes 2024-09-15T00:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3655056384 bytes 2024-09-15T00:38:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3723859968 bytes 2024-09-15T00:38:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3738440704 bytes 2024-09-15T00:38:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3809438720 bytes 2024-09-15T00:38:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3840431104 bytes 2024-09-15T00:38:42.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3894004736 bytes 2024-09-15T00:38:42.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3955027968 bytes 2024-09-15T00:38:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 3995836416 bytes 2024-09-15T00:38:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4031275008 bytes 2024-09-15T00:38:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4051458048 bytes 2024-09-15T00:38:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4096517120 bytes 2024-09-15T00:38:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4152593408 bytes 2024-09-15T00:38:43.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4165424128 bytes 2024-09-15T00:38:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4203451392 bytes 2024-09-15T00:38:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4329500672 bytes 2024-09-15T00:38:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4370527232 bytes 2024-09-15T00:38:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4413028352 bytes 2024-09-15T00:38:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4501861376 bytes 2024-09-15T00:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 4504565760 bytes 2024-09-15T00:38:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4514693120 bytes 2024-09-15T00:38:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 4545694720 bytes 2024-09-15T00:38:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 4600144896 bytes 2024-09-15T00:38:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 4641998848 bytes 2024-09-15T00:38:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11100 IOs, 4690147328 bytes 2024-09-15T00:38:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11200 IOs, 4743511040 bytes 2024-09-15T00:38:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11300 IOs, 4768331776 bytes 2024-09-15T00:38:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11400 IOs, 4809443328 bytes 2024-09-15T00:38:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11500 IOs, 4836082688 bytes 2024-09-15T00:38:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11600 IOs, 4860634112 bytes 2024-09-15T00:38:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11700 IOs, 4911165440 bytes 2024-09-15T00:38:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11800 IOs, 4954743808 bytes 2024-09-15T00:38:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11900 IOs, 4962845696 bytes 2024-09-15T00:38:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12000 IOs, 5018532864 bytes 2024-09-15T00:38:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12100 IOs, 5099543552 bytes 2024-09-15T00:38:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12200 IOs, 5100161024 bytes 2024-09-15T00:38:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12300 IOs, 5157846016 bytes 2024-09-15T00:38:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12400 IOs, 5211783168 bytes 2024-09-15T00:38:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12500 IOs, 5279332352 bytes 2024-09-15T00:38:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12600 IOs, 5290845184 bytes 2024-09-15T00:38:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12700 IOs, 5309400064 bytes 2024-09-15T00:38:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12800 IOs, 5310022656 bytes 2024-09-15T00:38:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12900 IOs, 5335089152 bytes 2024-09-15T00:38:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13000 IOs, 5367148544 bytes 2024-09-15T00:38:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13100 IOs, 5400765440 bytes 2024-09-15T00:38:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13200 IOs, 5462348800 bytes 2024-09-15T00:38:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13300 IOs, 5522621440 bytes 2024-09-15T00:38:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:13400 IOs, 5600829440 bytes 2024-09-15T00:38:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:38:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:38:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:38:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:38:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-15T00:38:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:38:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:38:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:38:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:38:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:38:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:38:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:38:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:38:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:38:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:38:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+replaying 2024-09-15T00:38:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: replaying, {"bytes_per_second":17772373.33,"bytes_per_snapshot":534815744.0,"last_snapshot_bytes":533171200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360704,"remote_snapshot_timestamp":1726360704,"replay_state":"idle"} 2024-09-15T00:38:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: service: mirror.1 on smithi059 2024-09-15T00:38:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:38:39 2024-09-15T00:38:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:38:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:38:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:38:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:38:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:38:22' 2024-09-15T00:38:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:38:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: state: up+replaying 2024-09-15T00:38:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ description: .*' 2024-09-15T00:38:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: description: replaying, {"bytes_per_second":17772373.33,"bytes_per_snapshot":534815744.0,"last_snapshot_bytes":533171200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360704,"remote_snapshot_timestamp":1726360704,"replay_state":"idle"} 2024-09-15T00:38:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:38:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:38:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^up+' 2024-09-15T00:38:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:38:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep 'service: *mirror..* on ' 2024-09-15T00:38:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: service: mirror.1 on smithi059 2024-09-15T00:38:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update 2024-09-15T00:38:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-15T00:38:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:38:39' 2024-09-15T00:38:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-15 00:38:39' 2024-09-15T00:38:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:38:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:38:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local 'prev_last_update=2024-09-15 00:38:39' 2024-09-15T00:38:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:38:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:38:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-15T00:38:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='OKOKOK142104293mirror.0smithi05919.1.1-324-gecab7be5falseOK42224295mirror.3smithi05919.1.1-324-gecab7be5falseOK42614264mirror.1smithi05919.1.1-324-gecab7be5trueOK43564362mirror.2smithi05919.1.1-324-gecab7be5falseOKtest23953c4f-e4d9-4660-9569-50c96f4cf0e9up+replayingreplaying, {"bytes_per_second":17772373.33,"bytes_per_snapshot":534815744.0,"last_snapshot_bytes":533171200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360704,"remote_snapshot_timestamp":1726360704,"replay_state":"idle"}42614264mirror.1smithi0592024-09-15 00:38:39cluster21108ad63-b7b4-4889-ae31-b1fd6de8431bup+stoppedlocal image is primary2024-09-15 00:38:22' 2024-09-15T00:38:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update state 2024-09-15T00:38:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-15T00:38:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:38:39' 2024-09-15T00:38:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-15T00:38:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=up+replaying 2024-09-15T00:38:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:38:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep up+replaying 2024-09-15T00:38:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:38:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:38:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:38:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap3 2024-09-15T00:38:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap3 2024-09-15T00:38:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:38:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:38:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:38:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap3 2024-09-15T00:38:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:38:57.792 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.426176839020149) > weight (0.25) 2024-09-15T00:38:57.792 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9615022715732601) > weight (0.25) 2024-09-15T00:38:57.793 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6426387217144036) > weight (0.25) 2024-09-15T00:38:57.793 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.2890365325368459) > weight (0.25) 2024-09-15T00:38:57.793 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-15T00:39:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:39:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:39:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:39:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:39:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:39:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:39:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:39:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:39:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:39:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:39:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:39:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:39:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:39:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:39:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:39:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:39:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:39:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:39:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:39:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:39:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:39:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:39:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:39:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:39:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:39:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:39:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:39:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:39:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:39:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:39:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:39:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:39:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:39:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:39:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:39:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:39:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:39:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:39:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:39:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:39:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:39:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:39:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:39:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:39:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:39:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:39:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:39:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:39:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:39:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:39:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:39:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:39:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:39:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:39:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:39:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:39:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:39:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:39:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:39:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:39:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:39:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:39:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:39:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:39:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:39:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:39:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 11 2024-09-15T00:39:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:39:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:39:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:39:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:39:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:39:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:39:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=11 2024-09-15T00:39:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:39:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:39:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:39:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:39:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:39:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:39:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:39:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:39:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:39:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:39:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:39:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:39:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:39:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 11 = '' 2024-09-15T00:39:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:39:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:39:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:39:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:39:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:39:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:39:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:39:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:39:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:39:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:39:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:39:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:39:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 11 = '' 2024-09-15T00:39:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:39:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:39:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:39:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:39:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:39:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:39:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:39:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:39:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:39:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:39:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:39:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:39:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 11 = '' 2024-09-15T00:39:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:39:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:39:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:39:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:39:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:39:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:39:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:39:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=11 2024-09-15T00:39:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 11 = 11 2024-09-15T00:39:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:39:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-15T00:39:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap3 2024-09-15T00:39:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:39:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:39:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:39:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-15T00:39:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:39:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:39:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:39:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 6 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:38:30 2024 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:39:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:39:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:39:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:39:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snap2-clean ']' 2024-09-15T00:39:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-15T00:39:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap2-clean 2024-09-15T00:39:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:39:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:39:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:39:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:39:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:39:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:39:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:39:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:39:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:39:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:39:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:39:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap3 2024-09-15T00:39:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap3 2024-09-15T00:39:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:39:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:39:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:39:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:39:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:39:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:39:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:39:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:39:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:39:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:39:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:39:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name=snap3-clean 2024-09-15T00:39:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-15T00:39:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:39:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap3-clean 2024-09-15T00:39:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-15T00:39:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:39:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror test 2024-09-15T00:39:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:39:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:39:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=13 2024-09-15T00:39:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:39:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-15T00:39:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 67426304 bytes 2024-09-15T00:39:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 161844224 bytes 2024-09-15T00:39:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 256300032 bytes 2024-09-15T00:39:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 310123520 bytes 2024-09-15T00:39:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 344957952 bytes 2024-09-15T00:39:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 397647872 bytes 2024-09-15T00:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 413295616 bytes 2024-09-15T00:39:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 446544896 bytes 2024-09-15T00:39:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 538790912 bytes 2024-09-15T00:39:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 558897152 bytes 2024-09-15T00:39:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 620598272 bytes 2024-09-15T00:39:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 661273600 bytes 2024-09-15T00:39:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 661878784 bytes 2024-09-15T00:39:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 730486784 bytes 2024-09-15T00:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 765138944 bytes 2024-09-15T00:39:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 791659520 bytes 2024-09-15T00:39:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 828235776 bytes 2024-09-15T00:39:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 842799104 bytes 2024-09-15T00:39:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 843393024 bytes 2024-09-15T00:39:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 954988544 bytes 2024-09-15T00:39:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1001829376 bytes 2024-09-15T00:39:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1050447872 bytes 2024-09-15T00:39:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1080900608 bytes 2024-09-15T00:39:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1110330368 bytes 2024-09-15T00:39:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1128183808 bytes 2024-09-15T00:39:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1134860288 bytes 2024-09-15T00:39:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1163616256 bytes 2024-09-15T00:39:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1198706688 bytes 2024-09-15T00:39:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1251209216 bytes 2024-09-15T00:39:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1265561600 bytes 2024-09-15T00:39:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1289496576 bytes 2024-09-15T00:39:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1332092928 bytes 2024-09-15T00:39:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1404680192 bytes 2024-09-15T00:39:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1442255872 bytes 2024-09-15T00:39:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1462848512 bytes 2024-09-15T00:39:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1516967936 bytes 2024-09-15T00:39:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1538512896 bytes 2024-09-15T00:39:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1589800960 bytes 2024-09-15T00:39:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1658270720 bytes 2024-09-15T00:39:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1691970560 bytes 2024-09-15T00:39:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1742068736 bytes 2024-09-15T00:39:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1791031296 bytes 2024-09-15T00:39:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1817853952 bytes 2024-09-15T00:39:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1880815616 bytes 2024-09-15T00:39:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1903639552 bytes 2024-09-15T00:39:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1927803904 bytes 2024-09-15T00:39:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1991815168 bytes 2024-09-15T00:39:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2018914304 bytes 2024-09-15T00:39:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2051107840 bytes 2024-09-15T00:39:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2074963968 bytes 2024-09-15T00:39:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2120456192 bytes 2024-09-15T00:39:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2125289472 bytes 2024-09-15T00:39:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2142980096 bytes 2024-09-15T00:39:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2185553920 bytes 2024-09-15T00:39:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2269678592 bytes 2024-09-15T00:39:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2292814848 bytes 2024-09-15T00:39:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2316433408 bytes 2024-09-15T00:39:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2339105792 bytes 2024-09-15T00:39:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2364646400 bytes 2024-09-15T00:39:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2385208320 bytes 2024-09-15T00:39:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2397572096 bytes 2024-09-15T00:39:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2497438720 bytes 2024-09-15T00:39:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2535685120 bytes 2024-09-15T00:39:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2553092096 bytes 2024-09-15T00:39:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2597276672 bytes 2024-09-15T00:39:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2627898368 bytes 2024-09-15T00:39:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2678685696 bytes 2024-09-15T00:39:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2730109952 bytes 2024-09-15T00:39:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2771338240 bytes 2024-09-15T00:39:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2771964928 bytes 2024-09-15T00:39:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2828679168 bytes 2024-09-15T00:39:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2877473792 bytes 2024-09-15T00:39:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2893658112 bytes 2024-09-15T00:39:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2901240832 bytes 2024-09-15T00:39:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2946558976 bytes 2024-09-15T00:39:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2979868672 bytes 2024-09-15T00:39:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3068716032 bytes 2024-09-15T00:39:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3125075968 bytes 2024-09-15T00:39:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3183288320 bytes 2024-09-15T00:39:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3227117568 bytes 2024-09-15T00:39:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3295470592 bytes 2024-09-15T00:39:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3337687040 bytes 2024-09-15T00:39:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3373070336 bytes 2024-09-15T00:39:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3400032256 bytes 2024-09-15T00:39:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3433326592 bytes 2024-09-15T00:39:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3454840832 bytes 2024-09-15T00:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3570066432 bytes 2024-09-15T00:39:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3618046976 bytes 2024-09-15T00:39:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3673327616 bytes 2024-09-15T00:39:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3725549568 bytes 2024-09-15T00:39:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3775815680 bytes 2024-09-15T00:39:39.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3856812032 bytes 2024-09-15T00:39:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3909343232 bytes 2024-09-15T00:39:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4000549888 bytes 2024-09-15T00:39:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4030124032 bytes 2024-09-15T00:39:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4054299648 bytes 2024-09-15T00:39:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4107415552 bytes 2024-09-15T00:39:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4177092608 bytes 2024-09-15T00:39:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4177710080 bytes 2024-09-15T00:39:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4201086976 bytes 2024-09-15T00:39:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4267902976 bytes 2024-09-15T00:39:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4287526912 bytes 2024-09-15T00:39:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4330798080 bytes 2024-09-15T00:39:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4347563008 bytes 2024-09-15T00:39:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4386761728 bytes 2024-09-15T00:39:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 4397989888 bytes 2024-09-15T00:39:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4451251200 bytes 2024-09-15T00:39:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 4482780160 bytes 2024-09-15T00:39:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 4514314240 bytes 2024-09-15T00:39:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 4548716544 bytes 2024-09-15T00:39:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11100 IOs, 4560928768 bytes 2024-09-15T00:39:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11200 IOs, 4619105280 bytes 2024-09-15T00:39:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11300 IOs, 4715390976 bytes 2024-09-15T00:39:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11400 IOs, 4767962112 bytes 2024-09-15T00:39:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11500 IOs, 4790156288 bytes 2024-09-15T00:39:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11600 IOs, 4833714176 bytes 2024-09-15T00:39:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11700 IOs, 4854849536 bytes 2024-09-15T00:39:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11800 IOs, 4890728448 bytes 2024-09-15T00:39:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11900 IOs, 4903453696 bytes 2024-09-15T00:39:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12000 IOs, 4925718528 bytes 2024-09-15T00:39:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12100 IOs, 4983613440 bytes 2024-09-15T00:39:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12200 IOs, 5035921408 bytes 2024-09-15T00:39:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12300 IOs, 5069641728 bytes 2024-09-15T00:39:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12400 IOs, 5123996672 bytes 2024-09-15T00:39:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12500 IOs, 5181947904 bytes 2024-09-15T00:39:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12600 IOs, 5242771456 bytes 2024-09-15T00:39:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12700 IOs, 5293135872 bytes 2024-09-15T00:39:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:39:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:39:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:39:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:39:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:39:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:39:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:39:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:39:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:39:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:39:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:39:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:39:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+replaying 2024-09-15T00:39:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: replaying, {"bytes_per_second":17869687.47,"bytes_per_snapshot":534630912.0,"last_snapshot_bytes":536090624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360760,"remote_snapshot_timestamp":1726360760,"replay_state":"idle"} 2024-09-15T00:39:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: service: mirror.1 on smithi059 2024-09-15T00:39:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:39:39 2024-09-15T00:39:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:39:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:39:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:39:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:39:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:39:22' 2024-09-15T00:39:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:39:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: state: up+replaying 2024-09-15T00:39:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ description: .*' 2024-09-15T00:39:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: description: replaying, {"bytes_per_second":17869687.47,"bytes_per_snapshot":534630912.0,"last_snapshot_bytes":536090624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360760,"remote_snapshot_timestamp":1726360760,"replay_state":"idle"} 2024-09-15T00:39:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:39:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:39:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^up+' 2024-09-15T00:39:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:39:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep 'service: *mirror..* on ' 2024-09-15T00:39:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: service: mirror.1 on smithi059 2024-09-15T00:39:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update 2024-09-15T00:39:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-15T00:39:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:39:39' 2024-09-15T00:39:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-15 00:39:39' 2024-09-15T00:39:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:39:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:39:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local 'prev_last_update=2024-09-15 00:39:39' 2024-09-15T00:39:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:39:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:39:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-15T00:39:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='OKOKOK142104293mirror.0smithi05919.1.1-324-gecab7be5falseOK42224295mirror.3smithi05919.1.1-324-gecab7be5falseOK42614264mirror.1smithi05919.1.1-324-gecab7be5trueOK43564362mirror.2smithi05919.1.1-324-gecab7be5falseOKtest23953c4f-e4d9-4660-9569-50c96f4cf0e9up+replayingreplaying, {"bytes_per_second":17869687.47,"bytes_per_snapshot":534630912.0,"last_snapshot_bytes":536090624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360760,"remote_snapshot_timestamp":1726360760,"replay_state":"idle"}42614264mirror.1smithi0592024-09-15 00:39:39cluster21108ad63-b7b4-4889-ae31-b1fd6de8431bup+stoppedlocal image is primary2024-09-15 00:39:22' 2024-09-15T00:39:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update state 2024-09-15T00:39:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-15T00:39:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:39:39' 2024-09-15T00:39:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-15T00:39:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=up+replaying 2024-09-15T00:39:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:39:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep up+replaying 2024-09-15T00:39:43.798 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6335389881880186) > weight (0.25) 2024-09-15T00:39:43.799 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9787997419038738) > weight (0.25) 2024-09-15T00:39:43.799 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7085334826144629) > weight (0.25) 2024-09-15T00:39:43.799 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6951728799294837) > weight (0.25) 2024-09-15T00:39:43.799 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-09-15T00:39:43.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:39:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:39:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:39:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap4 2024-09-15T00:39:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap4 2024-09-15T00:39:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:39:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:39:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:39:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap4 2024-09-15T00:39:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:40:05.794 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5400719243944331) > weight (0.25) 2024-09-15T00:40:05.795 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-15T00:40:05.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-15T00:40:05.795 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-15T00:40:05.796 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-15T00:40:05.796 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:40:05.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T00:40:05.795+0000 7f5d5bb8f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 36765) UID: 1000 2024-09-15T00:40:11.897 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-15T00:40:11.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-15T00:40:11.898 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-15T00:40:11.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-15T00:40:11.901 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-15T00:40:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:40:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:40:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:40:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:40:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:40:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:40:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:40:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:40:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:40:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:40:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:40:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:40:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:40:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:40:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:40:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:40:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:40:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:40:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:40:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:40:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:40:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:40:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:40:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:40:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:40:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:40:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:40:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:40:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:40:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:40:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:40:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:40:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:40:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:40:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:40:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:40:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:40:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:40:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:40:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:40:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:40:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:40:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:40:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:40:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:40:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:40:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:40:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:40:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:40:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:40:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:40:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:40:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:40:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:40:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:40:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:40:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:40:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:40:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:40:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:40:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:40:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:40:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 14 2024-09-15T00:40:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:40:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:40:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:40:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:40:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:40:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:40:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=14 2024-09-15T00:40:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:40:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:40:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:40:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:40:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:40:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:40:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:40:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:40:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:40:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:40:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:40:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:40:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:40:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 14 = '' 2024-09-15T00:40:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:40:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:40:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:40:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:40:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:40:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:40:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:40:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:40:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:40:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:40:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:40:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:40:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 14 = '' 2024-09-15T00:40:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:40:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:40:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:40:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:40:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:40:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:40:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:40:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:40:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:40:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:40:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:40:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:40:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 14 = '' 2024-09-15T00:40:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:40:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:40:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:40:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:40:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:40:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:40:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:40:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=14 2024-09-15T00:40:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 14 = 14 2024-09-15T00:40:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:40:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-15T00:40:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap4 2024-09-15T00:40:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:40:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:40:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:40:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-15T00:40:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:40:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:40:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:40:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 8 2024-09-15T00:40:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:38:30 2024 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:40:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:40:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:40:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snap3-clean ']' 2024-09-15T00:40:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-15T00:40:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap3-clean 2024-09-15T00:40:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:40:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:40:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:40:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:40:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:40:21.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:40:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:40:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:40:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:40:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:40:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:40:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap4 2024-09-15T00:40:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap4 2024-09-15T00:40:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:40:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:40:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:40:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:40:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:40:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:40:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:40:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:40:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:40:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:40:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:40:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name=snap4-clean 2024-09-15T00:40:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-15T00:40:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:40:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap4-clean 2024-09-15T00:40:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-15T00:40:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:40:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror test 2024-09-15T00:40:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:40:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=10 2024-09-15T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-15T00:40:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 30061568 bytes 2024-09-15T00:40:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 81804288 bytes 2024-09-15T00:40:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 173093888 bytes 2024-09-15T00:40:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 226678784 bytes 2024-09-15T00:40:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 306290688 bytes 2024-09-15T00:40:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 333383680 bytes 2024-09-15T00:40:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 398328832 bytes 2024-09-15T00:40:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 416452608 bytes 2024-09-15T00:40:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 476396544 bytes 2024-09-15T00:40:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 537312256 bytes 2024-09-15T00:40:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 577897472 bytes 2024-09-15T00:40:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 648169472 bytes 2024-09-15T00:40:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 688975872 bytes 2024-09-15T00:40:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 713073664 bytes 2024-09-15T00:40:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 780270592 bytes 2024-09-15T00:40:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 780863488 bytes 2024-09-15T00:40:26.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 820849664 bytes 2024-09-15T00:40:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 882594816 bytes 2024-09-15T00:40:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 884712448 bytes 2024-09-15T00:40:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 947231744 bytes 2024-09-15T00:40:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 969779200 bytes 2024-09-15T00:40:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1003105280 bytes 2024-09-15T00:40:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1042093056 bytes 2024-09-15T00:40:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1114045440 bytes 2024-09-15T00:40:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1160057856 bytes 2024-09-15T00:40:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1185242112 bytes 2024-09-15T00:40:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1195052032 bytes 2024-09-15T00:40:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1213848576 bytes 2024-09-15T00:40:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1237233664 bytes 2024-09-15T00:40:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1294633984 bytes 2024-09-15T00:40:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1373034496 bytes 2024-09-15T00:40:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1424683008 bytes 2024-09-15T00:40:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1472966656 bytes 2024-09-15T00:40:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1517163520 bytes 2024-09-15T00:40:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1550748672 bytes 2024-09-15T00:40:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1622727680 bytes 2024-09-15T00:40:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1638454272 bytes 2024-09-15T00:40:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1652251648 bytes 2024-09-15T00:40:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1652859904 bytes 2024-09-15T00:40:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1704480768 bytes 2024-09-15T00:40:28.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1735025664 bytes 2024-09-15T00:40:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1779309568 bytes 2024-09-15T00:40:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1817620480 bytes 2024-09-15T00:40:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1846684672 bytes 2024-09-15T00:40:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1883817984 bytes 2024-09-15T00:40:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1908041728 bytes 2024-09-15T00:40:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1908640768 bytes 2024-09-15T00:40:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1962401792 bytes 2024-09-15T00:40:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1994034176 bytes 2024-09-15T00:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1995128832 bytes 2024-09-15T00:40:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1996152832 bytes 2024-09-15T00:40:29.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2049577984 bytes 2024-09-15T00:40:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2105448448 bytes 2024-09-15T00:40:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2130439168 bytes 2024-09-15T00:40:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2208326656 bytes 2024-09-15T00:40:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2250760192 bytes 2024-09-15T00:40:30.145 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:40:30.143+0000 7fc49cacc640 -1 reset not still connected to 0x560c49952c30 2024-09-15T00:40:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2319095808 bytes 2024-09-15T00:40:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2372327424 bytes 2024-09-15T00:40:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2429778944 bytes 2024-09-15T00:40:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2544957440 bytes 2024-09-15T00:40:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2643098624 bytes 2024-09-15T00:40:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2673793024 bytes 2024-09-15T00:40:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2769285120 bytes 2024-09-15T00:40:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2794962944 bytes 2024-09-15T00:40:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2835992576 bytes 2024-09-15T00:40:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2860333056 bytes 2024-09-15T00:40:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3001762816 bytes 2024-09-15T00:40:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3044418560 bytes 2024-09-15T00:40:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3076551680 bytes 2024-09-15T00:40:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3164153856 bytes 2024-09-15T00:40:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3196877824 bytes 2024-09-15T00:40:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3291741184 bytes 2024-09-15T00:40:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3338673152 bytes 2024-09-15T00:40:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3357077504 bytes 2024-09-15T00:40:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3378021376 bytes 2024-09-15T00:40:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3414609920 bytes 2024-09-15T00:40:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3445102592 bytes 2024-09-15T00:40:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3487589376 bytes 2024-09-15T00:40:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3498620928 bytes 2024-09-15T00:40:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3531734016 bytes 2024-09-15T00:40:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3561706496 bytes 2024-09-15T00:40:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3615858688 bytes 2024-09-15T00:40:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3678322688 bytes 2024-09-15T00:40:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3717896192 bytes 2024-09-15T00:40:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3745543168 bytes 2024-09-15T00:40:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3810677760 bytes 2024-09-15T00:40:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3832953856 bytes 2024-09-15T00:40:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3872019456 bytes 2024-09-15T00:40:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3929972736 bytes 2024-09-15T00:40:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3965405184 bytes 2024-09-15T00:40:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3965999104 bytes 2024-09-15T00:40:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3990128640 bytes 2024-09-15T00:40:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 4076171264 bytes 2024-09-15T00:40:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:40:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:40:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:40:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:40:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-15T00:40:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:40:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:40:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:40:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:40:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:40:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:40:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:40:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:40:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:40:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:40:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:40:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:40:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:40:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:40:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+replaying 2024-09-15T00:40:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534630912.0,"last_snapshot_bytes":536090624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360760,"remote_snapshot_timestamp":1726360760,"replay_state":"idle"} 2024-09-15T00:40:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: service: mirror.1 on smithi059 2024-09-15T00:40:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:40:09 2024-09-15T00:40:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:40:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:40:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:40:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:40:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:40:22' 2024-09-15T00:40:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:40:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: state: up+replaying 2024-09-15T00:40:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ description: .*' 2024-09-15T00:40:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534630912.0,"last_snapshot_bytes":536090624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360760,"remote_snapshot_timestamp":1726360760,"replay_state":"idle"} 2024-09-15T00:40:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:40:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:40:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^up+' 2024-09-15T00:40:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:40:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep 'service: *mirror..* on ' 2024-09-15T00:40:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: service: mirror.1 on smithi059 2024-09-15T00:40:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update 2024-09-15T00:40:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-15T00:40:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:40:09' 2024-09-15T00:40:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-15 00:40:09' 2024-09-15T00:40:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:40:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:40:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local 'prev_last_update=2024-09-15 00:40:09' 2024-09-15T00:40:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-15T00:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='OKOKOK142104293mirror.0smithi05919.1.1-324-gecab7be5falseOK42224295mirror.3smithi05919.1.1-324-gecab7be5falseOK42614264mirror.1smithi05919.1.1-324-gecab7be5trueOK43564362mirror.2smithi05919.1.1-324-gecab7be5falseOKtest23953c4f-e4d9-4660-9569-50c96f4cf0e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534630912.0,"last_snapshot_bytes":536090624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360760,"remote_snapshot_timestamp":1726360760,"replay_state":"idle"}42614264mirror.1smithi0592024-09-15 00:40:09cluster21108ad63-b7b4-4889-ae31-b1fd6de8431bup+stoppedlocal image is primary2024-09-15 00:40:22' 2024-09-15T00:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update state 2024-09-15T00:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-15T00:40:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:40:09' 2024-09-15T00:40:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-15T00:40:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=up+replaying 2024-09-15T00:40:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:40:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep up+replaying 2024-09-15T00:40:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:40:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:40:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:40:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap5 2024-09-15T00:40:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap5 2024-09-15T00:40:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:40:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:40:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:40:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap5 2024-09-15T00:40:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:41:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:41:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:41:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:41:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:41:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:41:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:41:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:41:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:41:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:41:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:41:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:41:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:41:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:41:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:41:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:41:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:41:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:41:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:41:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:41:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:41:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:41:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:41:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:41:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:41:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:41:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:41:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:41:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:41:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:41:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:41:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:41:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:41:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:41:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:41:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:41:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:41:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:41:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:41:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:41:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:41:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:41:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:41:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:41:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:41:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:41:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:41:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:41:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:41:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:41:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:41:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:41:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:41:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:41:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:41:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:41:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:41:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:41:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:41:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:41:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 17 2024-09-15T00:41:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:41:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:41:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:41:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:41:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=17 2024-09-15T00:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:41:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:41:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:41:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:41:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:41:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:41:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:41:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:41:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:41:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 17 = '' 2024-09-15T00:41:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:41:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:41:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:41:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:41:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:41:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:41:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:41:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:41:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:41:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:41:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 17 = '' 2024-09-15T00:41:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:41:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:41:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:41:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:41:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:41:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:41:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:41:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:41:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:41:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 17 = '' 2024-09-15T00:41:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:41:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:41:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:41:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:41:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:41:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=17 2024-09-15T00:41:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 17 = 17 2024-09-15T00:41:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:41:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-15T00:41:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap5 2024-09-15T00:41:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:41:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:41:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:41:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-15T00:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 10 2024-09-15T00:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:40:21 2024 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:41:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:41:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snap4-clean ']' 2024-09-15T00:41:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-15T00:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap4-clean 2024-09-15T00:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:41:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:41:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:41:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:41:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:41:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:41:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:41:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap5 2024-09-15T00:41:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap5 2024-09-15T00:41:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:41:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:41:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:41:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:41:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:41:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:41:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:41:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:41:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:41:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:41:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:41:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name=snap5-clean 2024-09-15T00:41:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-15T00:41:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap5-clean 2024-09-15T00:41:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-15T00:41:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:41:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror test 2024-09-15T00:41:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:41:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=6 2024-09-15T00:41:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:41:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-15T00:41:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 42488832 bytes 2024-09-15T00:41:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 90192896 bytes 2024-09-15T00:41:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 120681472 bytes 2024-09-15T00:41:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 176489472 bytes 2024-09-15T00:41:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 206167040 bytes 2024-09-15T00:41:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 215189504 bytes 2024-09-15T00:41:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 272416768 bytes 2024-09-15T00:41:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 321726464 bytes 2024-09-15T00:41:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 375766016 bytes 2024-09-15T00:41:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 468517888 bytes 2024-09-15T00:41:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 524721152 bytes 2024-09-15T00:41:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 568499200 bytes 2024-09-15T00:41:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 602163200 bytes 2024-09-15T00:41:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 602767360 bytes 2024-09-15T00:41:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 619040768 bytes 2024-09-15T00:41:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 686910464 bytes 2024-09-15T00:41:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 708325376 bytes 2024-09-15T00:41:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 759013376 bytes 2024-09-15T00:41:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 808954880 bytes 2024-09-15T00:41:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 860475392 bytes 2024-09-15T00:41:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 962813952 bytes 2024-09-15T00:41:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 997795840 bytes 2024-09-15T00:41:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1033519104 bytes 2024-09-15T00:41:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1103606784 bytes 2024-09-15T00:41:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1141055488 bytes 2024-09-15T00:41:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1189335040 bytes 2024-09-15T00:41:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1224841216 bytes 2024-09-15T00:41:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1244840960 bytes 2024-09-15T00:41:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1259566080 bytes 2024-09-15T00:41:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1287822336 bytes 2024-09-15T00:41:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1310198784 bytes 2024-09-15T00:41:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1342026752 bytes 2024-09-15T00:41:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1349030912 bytes 2024-09-15T00:41:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1415727104 bytes 2024-09-15T00:41:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1449746432 bytes 2024-09-15T00:41:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1471685632 bytes 2024-09-15T00:41:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1541227520 bytes 2024-09-15T00:41:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1597493248 bytes 2024-09-15T00:41:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1622333440 bytes 2024-09-15T00:41:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1704558592 bytes 2024-09-15T00:41:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1724876800 bytes 2024-09-15T00:41:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1741011968 bytes 2024-09-15T00:41:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1784148992 bytes 2024-09-15T00:41:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1828938752 bytes 2024-09-15T00:41:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1883146240 bytes 2024-09-15T00:41:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1962756096 bytes 2024-09-15T00:41:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1977133056 bytes 2024-09-15T00:41:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2029148160 bytes 2024-09-15T00:41:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2058866688 bytes 2024-09-15T00:41:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2103081984 bytes 2024-09-15T00:41:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2159542272 bytes 2024-09-15T00:41:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2211836928 bytes 2024-09-15T00:41:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2239237120 bytes 2024-09-15T00:41:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2270503936 bytes 2024-09-15T00:41:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:41:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:41:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:41:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:41:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-15T00:41:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:41:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:41:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:41:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:41:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:41:22.902 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3916771662346151) > weight (0.25) 2024-09-15T00:41:22.903 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5865848618038897) > weight (0.25) 2024-09-15T00:41:22.903 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7150288568657244) > weight (0.25) 2024-09-15T00:41:22.903 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-15T00:41:22.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-15T00:41:22.904 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-15T00:41:22.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T00:41:22.903+0000 7f021b6b5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 36845) UID: 1000 2024-09-15T00:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:41:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:41:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:41:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:41:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:41:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:41:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:41:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:41:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:41:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+replaying 2024-09-15T00:41:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536047104.0,"last_snapshot_bytes":536003584,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360814,"remote_snapshot_timestamp":1726360814,"replay_state":"idle"} 2024-09-15T00:41:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: service: mirror.1 on smithi059 2024-09-15T00:41:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:41:09 2024-09-15T00:41:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:41:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:41:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:41:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:41:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:41:22' 2024-09-15T00:41:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:41:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: state: up+replaying 2024-09-15T00:41:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ description: .*' 2024-09-15T00:41:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536047104.0,"last_snapshot_bytes":536003584,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360814,"remote_snapshot_timestamp":1726360814,"replay_state":"idle"} 2024-09-15T00:41:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:41:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:41:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^up+' 2024-09-15T00:41:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:41:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep 'service: *mirror..* on ' 2024-09-15T00:41:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: service: mirror.1 on smithi059 2024-09-15T00:41:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update 2024-09-15T00:41:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-15T00:41:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:41:09' 2024-09-15T00:41:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-15 00:41:09' 2024-09-15T00:41:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:41:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local 'prev_last_update=2024-09-15 00:41:09' 2024-09-15T00:41:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:41:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:41:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-15T00:41:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='OKOKOK142104293mirror.0smithi05919.1.1-324-gecab7be5falseOK42224295mirror.3smithi05919.1.1-324-gecab7be5falseOK42614264mirror.1smithi05919.1.1-324-gecab7be5trueOK43564362mirror.2smithi05919.1.1-324-gecab7be5falseOKtest23953c4f-e4d9-4660-9569-50c96f4cf0e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536047104.0,"last_snapshot_bytes":536003584,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360814,"remote_snapshot_timestamp":1726360814,"replay_state":"idle"}42614264mirror.1smithi0592024-09-15 00:41:09cluster21108ad63-b7b4-4889-ae31-b1fd6de8431bup+stoppedlocal image is primary2024-09-15 00:41:22' 2024-09-15T00:41:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update state 2024-09-15T00:41:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-15T00:41:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:41:09' 2024-09-15T00:41:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-15T00:41:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=up+replaying 2024-09-15T00:41:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:41:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep up+replaying 2024-09-15T00:41:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:41:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:41:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:41:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap6 2024-09-15T00:41:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap6 2024-09-15T00:41:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap6 2024-09-15T00:41:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:41:26.800 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-15T00:41:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-15T00:41:26.801 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-15T00:41:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T00:41:26.800+0000 7f45ef18f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 36565) UID: 1000 2024-09-15T00:41:31.904 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-15T00:41:31.904 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:41:31.905 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-15T00:41:31.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-15T00:41:31.905 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-15T00:41:31.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-15T00:41:31.909 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-09-15T00:41:34.802 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-15T00:41:34.802 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:41:34.802 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-15T00:41:34.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-15T00:41:34.803 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-15T00:41:34.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-15T00:41:34.806 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-15T00:41:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:41:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:41:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:41:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:41:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:41:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:41:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:41:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:41:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:41:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:41:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:41:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:41:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:41:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:41:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:41:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:41:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:41:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:41:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:41:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:41:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:41:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:41:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:41:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:41:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:41:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:41:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:41:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:41:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:41:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:41:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:41:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:41:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:41:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:41:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:41:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:41:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:41:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:41:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:41:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:41:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:41:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:41:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:41:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:41:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:41:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:41:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:41:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:41:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:41:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:41:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:41:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:41:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:41:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:41:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:41:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:41:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:41:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:41:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:41:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:41:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:41:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:41:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:41:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:41:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:41:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:41:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:41:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:41:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:41:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:41:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:41:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:41:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:41:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:41:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:41:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:41:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:41:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 20 2024-09-15T00:41:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:41:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:41:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:41:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:41:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:41:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:41:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=20 2024-09-15T00:41:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:41:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:41:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:41:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:41:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:41:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:41:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:41:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:41:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:41:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:41:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:41:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 20 = '' 2024-09-15T00:41:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:41:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:41:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:41:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:41:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:41:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:41:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:41:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:41:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:41:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:41:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 20 = '' 2024-09-15T00:41:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:41:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:41:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:41:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:41:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:41:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:41:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:41:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:41:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:41:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:41:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:41:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:41:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:41:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:41:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 20 = '' 2024-09-15T00:41:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:41:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:42:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:42:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:42:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=20 2024-09-15T00:42:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 20 = 20 2024-09-15T00:42:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-15T00:42:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap6 2024-09-15T00:42:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:42:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:42:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:42:01.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-15T00:42:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:42:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:42:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:42:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 12 2024-09-15T00:42:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:42:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:42:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:40:21 2024 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:42:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:42:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snap5-clean ']' 2024-09-15T00:42:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-15T00:42:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap5-clean 2024-09-15T00:42:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:42:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:42:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:42:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:42:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:42:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:02.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap6 2024-09-15T00:42:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap6 2024-09-15T00:42:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:42:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:42:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:42:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:42:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:42:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name=snap6-clean 2024-09-15T00:42:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-15T00:42:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:42:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap6-clean 2024-09-15T00:42:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-15T00:42:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:42:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror test 2024-09-15T00:42:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:42:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:42:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=5 2024-09-15T00:42:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:42:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-15T00:42:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 31288320 bytes 2024-09-15T00:42:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 124634112 bytes 2024-09-15T00:42:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 125237248 bytes 2024-09-15T00:42:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 130608128 bytes 2024-09-15T00:42:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 184983552 bytes 2024-09-15T00:42:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 204430336 bytes 2024-09-15T00:42:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 223489024 bytes 2024-09-15T00:42:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 224105472 bytes 2024-09-15T00:42:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 266030080 bytes 2024-09-15T00:42:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 349591552 bytes 2024-09-15T00:42:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 418864128 bytes 2024-09-15T00:42:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 419490816 bytes 2024-09-15T00:42:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 471033856 bytes 2024-09-15T00:42:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 511714304 bytes 2024-09-15T00:42:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 577497088 bytes 2024-09-15T00:42:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 624614400 bytes 2024-09-15T00:42:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 727988224 bytes 2024-09-15T00:42:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 769397760 bytes 2024-09-15T00:42:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 822315008 bytes 2024-09-15T00:42:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 884354048 bytes 2024-09-15T00:42:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 935476224 bytes 2024-09-15T00:42:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 975395840 bytes 2024-09-15T00:42:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1012049920 bytes 2024-09-15T00:42:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1055205376 bytes 2024-09-15T00:42:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1123056640 bytes 2024-09-15T00:42:07.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1187549184 bytes 2024-09-15T00:42:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1218706432 bytes 2024-09-15T00:42:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1230968832 bytes 2024-09-15T00:42:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1282257920 bytes 2024-09-15T00:42:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1325520896 bytes 2024-09-15T00:42:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1393944576 bytes 2024-09-15T00:42:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1436563456 bytes 2024-09-15T00:42:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1454377984 bytes 2024-09-15T00:42:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1514949632 bytes 2024-09-15T00:42:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1530287104 bytes 2024-09-15T00:42:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1609530368 bytes 2024-09-15T00:42:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1614404608 bytes 2024-09-15T00:42:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1629281280 bytes 2024-09-15T00:42:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1699388416 bytes 2024-09-15T00:42:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1726477312 bytes 2024-09-15T00:42:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1758155776 bytes 2024-09-15T00:42:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1807287296 bytes 2024-09-15T00:42:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1829100544 bytes 2024-09-15T00:42:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1852162048 bytes 2024-09-15T00:42:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:42:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:42:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:42:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:42:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:42:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+replaying 2024-09-15T00:42:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: replaying, {"bytes_per_second":17492718.93,"bytes_per_snapshot":530547200.0,"last_snapshot_bytes":524781568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360915,"remote_snapshot_timestamp":1726360915,"replay_state":"idle"} 2024-09-15T00:42:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: service: mirror.1 on smithi059 2024-09-15T00:42:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:42:09 2024-09-15T00:42:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:42:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:42:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:42:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:42:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:41:52' 2024-09-15T00:42:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:42:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: state: up+replaying 2024-09-15T00:42:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ description: .*' 2024-09-15T00:42:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: description: replaying, {"bytes_per_second":17492718.93,"bytes_per_snapshot":530547200.0,"last_snapshot_bytes":524781568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360915,"remote_snapshot_timestamp":1726360915,"replay_state":"idle"} 2024-09-15T00:42:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:42:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:42:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^up+' 2024-09-15T00:42:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:42:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep 'service: *mirror..* on ' 2024-09-15T00:42:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: service: mirror.1 on smithi059 2024-09-15T00:42:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update 2024-09-15T00:42:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-15T00:42:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:42:09' 2024-09-15T00:42:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-15 00:42:09' 2024-09-15T00:42:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:42:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local 'prev_last_update=2024-09-15 00:42:09' 2024-09-15T00:42:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:42:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:42:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-15T00:42:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='OKOKOK142224295mirror.3smithi05919.1.1-324-gecab7be5falseOK42614264mirror.1smithi05919.1.1-324-gecab7be5trueOK43564362mirror.2smithi05919.1.1-324-gecab7be5falseOK44794482mirror.0smithi05919.1.1-324-gecab7be5falseOKtest23953c4f-e4d9-4660-9569-50c96f4cf0e9up+replayingreplaying, {"bytes_per_second":17492718.93,"bytes_per_snapshot":530547200.0,"last_snapshot_bytes":524781568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360915,"remote_snapshot_timestamp":1726360915,"replay_state":"idle"}42614264mirror.1smithi0592024-09-15 00:42:09cluster21108ad63-b7b4-4889-ae31-b1fd6de8431bup+stoppedlocal image is primary2024-09-15 00:41:52' 2024-09-15T00:42:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update state 2024-09-15T00:42:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-15T00:42:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:42:09' 2024-09-15T00:42:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-15T00:42:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=up+replaying 2024-09-15T00:42:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:42:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep up+replaying 2024-09-15T00:42:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:42:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap7 2024-09-15T00:42:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap7 2024-09-15T00:42:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:42:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap7 2024-09-15T00:42:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:42:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:42:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:42:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:42:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:42:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:42:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:42:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:42:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:42:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:42:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:42:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:42:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:42:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:42:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:42:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:42:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:42:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:42:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:42:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:42:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:42:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:42:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:42:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:42:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:42:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:42:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:42:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:42:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:42:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:42:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:42:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:42:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:42:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:42:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:42:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:42:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:42:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:42:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:42:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:42:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:42:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:42:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:42:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:42:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:42:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:42:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:42:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:42:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:42:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:42:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:42:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:42:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:42:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:42:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:42:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:42:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:42:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:42:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:42:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:42:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:42:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:42:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:42:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:42:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:42:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:42:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:42:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:42:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:42:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:42:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:42:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:42:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:42:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:42:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:42:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 23 2024-09-15T00:42:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:42:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:42:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:42:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:42:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:42:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:42:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=23 2024-09-15T00:42:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:42:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:42:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:42:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:42:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:42:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:42:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:42:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:42:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:42:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:42:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:42:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 23 = '' 2024-09-15T00:42:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:42:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:42:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:42:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:42:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:42:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:42:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:42:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:42:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:42:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:42:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 23 = '' 2024-09-15T00:42:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:42:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:42:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:42:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:42:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:42:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:42:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:42:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:42:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:42:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:42:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 23 = '' 2024-09-15T00:42:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:42:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:42:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:42:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:42:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:42:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:42:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=23 2024-09-15T00:42:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 23 = 23 2024-09-15T00:42:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-15T00:42:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:42:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap7 2024-09-15T00:42:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:42:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:42:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:42:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 14 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:42:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:42:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:42:01 2024 2024-09-15T00:42:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:42:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:42:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:42:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:42:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:42:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:42:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snap6-clean ']' 2024-09-15T00:42:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-15T00:42:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap6-clean 2024-09-15T00:42:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:42:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:42:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:42:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:42:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:42:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap7 2024-09-15T00:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap7 2024-09-15T00:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:42:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:42:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:42:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:42:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:42:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name=snap7-clean 2024-09-15T00:42:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-15T00:42:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:42:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap7-clean 2024-09-15T00:42:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-15T00:42:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:42:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror test 2024-09-15T00:42:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:42:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:42:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:42:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:42:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=13 2024-09-15T00:42:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:42:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-15T00:42:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 57863168 bytes 2024-09-15T00:42:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 151161856 bytes 2024-09-15T00:42:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 237052928 bytes 2024-09-15T00:42:51.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 268582912 bytes 2024-09-15T00:42:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 323073024 bytes 2024-09-15T00:42:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 366548992 bytes 2024-09-15T00:42:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 388527104 bytes 2024-09-15T00:42:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 465152000 bytes 2024-09-15T00:42:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 505296896 bytes 2024-09-15T00:42:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 573506560 bytes 2024-09-15T00:42:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 655900672 bytes 2024-09-15T00:42:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 688491520 bytes 2024-09-15T00:42:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 731905024 bytes 2024-09-15T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 745229312 bytes 2024-09-15T00:42:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 780164096 bytes 2024-09-15T00:42:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 795998208 bytes 2024-09-15T00:42:52.910 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8665629590233156) > weight (0.25) 2024-09-15T00:42:52.910 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-15T00:42:52.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-15T00:42:52.911 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-15T00:42:52.911 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-15T00:42:52.911 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:42:52.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T00:42:52.910+0000 7f5ccf7be640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 40013) UID: 1000 2024-09-15T00:42:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 854172672 bytes 2024-09-15T00:42:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 906516480 bytes 2024-09-15T00:42:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 968085504 bytes 2024-09-15T00:42:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1011470336 bytes 2024-09-15T00:42:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1034865664 bytes 2024-09-15T00:42:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1045715968 bytes 2024-09-15T00:42:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1053582336 bytes 2024-09-15T00:42:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1054812160 bytes 2024-09-15T00:42:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1122106368 bytes 2024-09-15T00:42:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1156404224 bytes 2024-09-15T00:42:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1189895168 bytes 2024-09-15T00:42:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1192563712 bytes 2024-09-15T00:42:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1238555648 bytes 2024-09-15T00:42:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1285536768 bytes 2024-09-15T00:42:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1306764288 bytes 2024-09-15T00:42:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1327764480 bytes 2024-09-15T00:42:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1419637760 bytes 2024-09-15T00:42:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1492683776 bytes 2024-09-15T00:42:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1542002688 bytes 2024-09-15T00:42:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1577342976 bytes 2024-09-15T00:42:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1588149248 bytes 2024-09-15T00:42:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1624259584 bytes 2024-09-15T00:42:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1664622592 bytes 2024-09-15T00:42:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1709208576 bytes 2024-09-15T00:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1728736256 bytes 2024-09-15T00:42:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1792978944 bytes 2024-09-15T00:42:55.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1810374656 bytes 2024-09-15T00:42:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1875173376 bytes 2024-09-15T00:42:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1938435072 bytes 2024-09-15T00:42:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2045872128 bytes 2024-09-15T00:42:56.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2071311360 bytes 2024-09-15T00:42:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2101488640 bytes 2024-09-15T00:42:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2143486976 bytes 2024-09-15T00:42:56.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2177008640 bytes 2024-09-15T00:42:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2217866240 bytes 2024-09-15T00:42:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2249534464 bytes 2024-09-15T00:42:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2286798848 bytes 2024-09-15T00:42:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2319678464 bytes 2024-09-15T00:42:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2347331584 bytes 2024-09-15T00:42:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2393076736 bytes 2024-09-15T00:42:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2437293056 bytes 2024-09-15T00:42:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2456669184 bytes 2024-09-15T00:42:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2500656128 bytes 2024-09-15T00:42:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2526347264 bytes 2024-09-15T00:42:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2562736128 bytes 2024-09-15T00:42:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2675308544 bytes 2024-09-15T00:42:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2746993664 bytes 2024-09-15T00:42:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2775362560 bytes 2024-09-15T00:42:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2798398464 bytes 2024-09-15T00:42:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2866270208 bytes 2024-09-15T00:42:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2927594496 bytes 2024-09-15T00:42:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2979999744 bytes 2024-09-15T00:42:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3015839744 bytes 2024-09-15T00:42:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3073872896 bytes 2024-09-15T00:42:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3102243840 bytes 2024-09-15T00:42:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3144732672 bytes 2024-09-15T00:42:59.013 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-15T00:42:59.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-15T00:42:59.013 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-15T00:42:59.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-15T00:42:59.016 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-15T00:42:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3152632832 bytes 2024-09-15T00:42:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3205348352 bytes 2024-09-15T00:42:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3233529856 bytes 2024-09-15T00:42:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3270387712 bytes 2024-09-15T00:42:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3334322176 bytes 2024-09-15T00:42:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3376237568 bytes 2024-09-15T00:42:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3476761600 bytes 2024-09-15T00:43:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3540410368 bytes 2024-09-15T00:43:00.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3611145216 bytes 2024-09-15T00:43:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3683369984 bytes 2024-09-15T00:43:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3706758144 bytes 2024-09-15T00:43:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3719244800 bytes 2024-09-15T00:43:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3750174720 bytes 2024-09-15T00:43:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3799181312 bytes 2024-09-15T00:43:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3838841856 bytes 2024-09-15T00:43:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3868734464 bytes 2024-09-15T00:43:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3920300032 bytes 2024-09-15T00:43:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3976341504 bytes 2024-09-15T00:43:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 4005965824 bytes 2024-09-15T00:43:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 4044217344 bytes 2024-09-15T00:43:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 4078660608 bytes 2024-09-15T00:43:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4106409984 bytes 2024-09-15T00:43:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4190180352 bytes 2024-09-15T00:43:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4275562496 bytes 2024-09-15T00:43:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4413090816 bytes 2024-09-15T00:43:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4495289344 bytes 2024-09-15T00:43:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4567775232 bytes 2024-09-15T00:43:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4597080064 bytes 2024-09-15T00:43:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4642762752 bytes 2024-09-15T00:43:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4698699776 bytes 2024-09-15T00:43:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4780127232 bytes 2024-09-15T00:43:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4821001216 bytes 2024-09-15T00:43:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4839633920 bytes 2024-09-15T00:43:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 4892631040 bytes 2024-09-15T00:43:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4981389312 bytes 2024-09-15T00:43:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 4990071808 bytes 2024-09-15T00:43:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 5034577920 bytes 2024-09-15T00:43:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 5077421056 bytes 2024-09-15T00:43:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:43:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:43:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:43:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-15T00:43:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:43:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:43:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:43:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:43:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:43:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:43:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:43:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+replaying 2024-09-15T00:43:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530547200.0,"last_snapshot_bytes":524781568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360915,"remote_snapshot_timestamp":1726360915,"replay_state":"idle"} 2024-09-15T00:43:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: service: mirror.1 on smithi059 2024-09-15T00:43:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:42:39 2024-09-15T00:43:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:43:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:43:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:43:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:43:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:42:52' 2024-09-15T00:43:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:43:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: state: up+replaying 2024-09-15T00:43:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ description: .*' 2024-09-15T00:43:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530547200.0,"last_snapshot_bytes":524781568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360915,"remote_snapshot_timestamp":1726360915,"replay_state":"idle"} 2024-09-15T00:43:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:43:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:43:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^up+' 2024-09-15T00:43:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:43:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep 'service: *mirror..* on ' 2024-09-15T00:43:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: service: mirror.1 on smithi059 2024-09-15T00:43:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update 2024-09-15T00:43:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-15T00:43:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:42:39' 2024-09-15T00:43:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-15 00:42:39' 2024-09-15T00:43:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:43:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local 'prev_last_update=2024-09-15 00:42:39' 2024-09-15T00:43:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:43:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:43:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-15T00:43:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='OKOKOK142224295mirror.3smithi05919.1.1-324-gecab7be5falseOK42614264mirror.1smithi05919.1.1-324-gecab7be5trueOK43564362mirror.2smithi05919.1.1-324-gecab7be5falseOK44794482mirror.0smithi05919.1.1-324-gecab7be5falseOKtest23953c4f-e4d9-4660-9569-50c96f4cf0e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530547200.0,"last_snapshot_bytes":524781568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360915,"remote_snapshot_timestamp":1726360915,"replay_state":"idle"}42614264mirror.1smithi0592024-09-15 00:42:39cluster21108ad63-b7b4-4889-ae31-b1fd6de8431bup+stoppedlocal image is primary2024-09-15 00:42:52' 2024-09-15T00:43:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update state 2024-09-15T00:43:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-15T00:43:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:42:39' 2024-09-15T00:43:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-15T00:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=up+replaying 2024-09-15T00:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep up+replaying 2024-09-15T00:43:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:43:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap8 2024-09-15T00:43:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap8 2024-09-15T00:43:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:43:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap8 2024-09-15T00:43:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:43:31.806 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.45251520868210526) > weight (0.25) 2024-09-15T00:43:31.807 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-15T00:43:31.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-15T00:43:31.808 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-15T00:43:31.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T00:43:31.807+0000 7f4417ad2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 37609) UID: 1000 2024-09-15T00:43:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:43:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:43:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:43:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:43:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:43:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:43:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:43:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:43:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:43:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:43:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:43:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:43:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:43:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:43:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:43:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:43:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:43:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:43:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:43:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:43:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:43:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:43:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:43:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:43:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:43:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:43:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:43:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:43:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:43:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:43:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:43:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:43:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:43:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:43:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:43:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:43:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:43:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:43:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:43:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:43:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:43:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:43:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:43:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:43:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:43:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:43:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-15T00:43:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:43:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-15T00:43:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:43:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:43:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:43:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:43:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:43:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:43:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:43:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-15T00:43:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:43:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:43:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-15T00:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:3 2024-09-15T00:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-15T00:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:3 2024-09-15T00:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=3 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=3 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=3 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:3 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:3 2024-09-15T00:43:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:43:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:43:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=3 2024-09-15T00:43:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-15T00:43:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:43:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:43:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=3 2024-09-15T00:43:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=3 2024-09-15T00:43:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-15T00:43:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:43:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:43:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:43:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-15T00:43:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:43:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:43:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:43:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:43:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 1 2024-09-15T00:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:43:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:43:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:43:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:43:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:43:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:43:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:43:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:43:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:43:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:43:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-15T00:43:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:43:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:43:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:43:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:43:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:43:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:43:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:43:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:43:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:43:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:43:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:43:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:43:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:43:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:43:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:43:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:43:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:43:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:43:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:43:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:43:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:43:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:43:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:43:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:43:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:43:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:43:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:43:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:43:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:43:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:43:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:43:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-15T00:43:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-15T00:43:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:43:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:43:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:43:35.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:43:35.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:43:35.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:43:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:43:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:43:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:43:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-15T00:43:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:43:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:43:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:43:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:43:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:3 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:3 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=3 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=3 2024-09-15T00:43:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=3 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:3 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:3 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=3 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:43:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=3 2024-09-15T00:43:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=3 2024-09-15T00:43:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-15T00:43:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:43:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:43:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-15T00:43:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-15T00:43:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-15T00:43:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:43:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:43:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:43:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 2 2024-09-15T00:43:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:43:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:43:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:43:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:43:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:43:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:43:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:43:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:43:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:43:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:43:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:43:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:43:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:43:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:43:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:43:37.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:43:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:43:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:43:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:43:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:43:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:43:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:43:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:43:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:43:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:43:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:43:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:43:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:43:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:43:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:43:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:43:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:43:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:43:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:43:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:43:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:43:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:43:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:43:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:43:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:43:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:43:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 26 2024-09-15T00:43:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:43:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:43:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:43:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:43:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:43:39.808 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-15T00:43:39.809 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:43:39.809 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-15T00:43:39.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-15T00:43:39.810 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-15T00:43:39.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-15T00:43:39.813 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-15T00:43:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:43:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=26 2024-09-15T00:43:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:43:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:43:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:43:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:43:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:43:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:43:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:43:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:43:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:43:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:43:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:43:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 26 = '' 2024-09-15T00:43:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:43:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:43:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:43:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:43:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:43:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:43:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:43:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:43:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:43:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:43:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:43:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:43:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 26 = '' 2024-09-15T00:43:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:43:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:43:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:43:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:43:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:43:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:43:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:43:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:43:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:43:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:43:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:43:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:43:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 26 = '' 2024-09-15T00:43:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:43:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:43:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:43:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:43:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:43:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:43:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=26 2024-09-15T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 26 = 26 2024-09-15T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-15T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap8 2024-09-15T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:43:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-15T00:43:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:43:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:43:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:43:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 16 2024-09-15T00:43:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:43:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:43:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:43:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:43:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:43:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:43:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:43:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:42:01 2024 2024-09-15T00:43:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:43:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:43:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:43:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:43:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:43:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:43:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snap7-clean ']' 2024-09-15T00:43:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-15T00:43:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap7-clean 2024-09-15T00:43:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:43:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:43:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:43:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:43:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:43:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:43:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:43:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:43:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:43:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:43:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap8 2024-09-15T00:43:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap8 2024-09-15T00:43:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:43:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:43:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:43:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:43:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:43:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:43:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:43:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:43:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:43:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name=snap8-clean 2024-09-15T00:43:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-15T00:43:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:43:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap8-clean 2024-09-15T00:43:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-15T00:43:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:43:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror test 2024-09-15T00:43:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:43:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:43:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=11 2024-09-15T00:43:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:43:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-15T00:43:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 49399808 bytes 2024-09-15T00:43:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 98548736 bytes 2024-09-15T00:43:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 118337536 bytes 2024-09-15T00:43:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 155481088 bytes 2024-09-15T00:43:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 216139776 bytes 2024-09-15T00:43:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 238105600 bytes 2024-09-15T00:43:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 358894592 bytes 2024-09-15T00:43:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 421581824 bytes 2024-09-15T00:43:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 484802560 bytes 2024-09-15T00:43:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 522836992 bytes 2024-09-15T00:43:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 580780032 bytes 2024-09-15T00:43:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 668318720 bytes 2024-09-15T00:43:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 800517120 bytes 2024-09-15T00:43:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 908422144 bytes 2024-09-15T00:43:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 960064512 bytes 2024-09-15T00:43:50.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 975214592 bytes 2024-09-15T00:43:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 975805440 bytes 2024-09-15T00:43:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 1023083520 bytes 2024-09-15T00:43:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 1049097216 bytes 2024-09-15T00:43:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1090395136 bytes 2024-09-15T00:43:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1126641664 bytes 2024-09-15T00:43:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1187273728 bytes 2024-09-15T00:43:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1299280896 bytes 2024-09-15T00:43:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1362258944 bytes 2024-09-15T00:43:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1398822912 bytes 2024-09-15T00:43:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1471418368 bytes 2024-09-15T00:43:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1482281984 bytes 2024-09-15T00:43:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1501274112 bytes 2024-09-15T00:43:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1501915136 bytes 2024-09-15T00:43:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1544433664 bytes 2024-09-15T00:43:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1594473472 bytes 2024-09-15T00:43:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1614105600 bytes 2024-09-15T00:43:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1650449408 bytes 2024-09-15T00:43:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1666472960 bytes 2024-09-15T00:43:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1731926016 bytes 2024-09-15T00:43:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1773038592 bytes 2024-09-15T00:43:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1826816000 bytes 2024-09-15T00:43:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1844548608 bytes 2024-09-15T00:43:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1921198080 bytes 2024-09-15T00:43:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1923794944 bytes 2024-09-15T00:43:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 2001844224 bytes 2024-09-15T00:43:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 2041094144 bytes 2024-09-15T00:43:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2053756928 bytes 2024-09-15T00:43:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2094410752 bytes 2024-09-15T00:43:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2125522944 bytes 2024-09-15T00:43:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2182610944 bytes 2024-09-15T00:43:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2262964224 bytes 2024-09-15T00:43:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2292843520 bytes 2024-09-15T00:43:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2393977856 bytes 2024-09-15T00:43:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2465395712 bytes 2024-09-15T00:43:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2466016256 bytes 2024-09-15T00:43:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2490240000 bytes 2024-09-15T00:43:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2510003200 bytes 2024-09-15T00:43:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2529898496 bytes 2024-09-15T00:43:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2611591168 bytes 2024-09-15T00:43:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2644546560 bytes 2024-09-15T00:43:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2714814464 bytes 2024-09-15T00:43:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2751943680 bytes 2024-09-15T00:43:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2809731072 bytes 2024-09-15T00:43:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2810332160 bytes 2024-09-15T00:43:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2860736512 bytes 2024-09-15T00:43:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2895200256 bytes 2024-09-15T00:43:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2933753856 bytes 2024-09-15T00:43:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2954393600 bytes 2024-09-15T00:43:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2965582848 bytes 2024-09-15T00:43:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 3033730048 bytes 2024-09-15T00:43:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3064940544 bytes 2024-09-15T00:43:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3107847168 bytes 2024-09-15T00:43:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3193185280 bytes 2024-09-15T00:43:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3233622016 bytes 2024-09-15T00:43:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3266888704 bytes 2024-09-15T00:43:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3319809024 bytes 2024-09-15T00:43:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3325410304 bytes 2024-09-15T00:43:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3394742272 bytes 2024-09-15T00:43:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3443330048 bytes 2024-09-15T00:43:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3464738816 bytes 2024-09-15T00:43:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3593662464 bytes 2024-09-15T00:43:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3635257344 bytes 2024-09-15T00:43:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3658419200 bytes 2024-09-15T00:43:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3679210496 bytes 2024-09-15T00:43:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3732886528 bytes 2024-09-15T00:43:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3791994880 bytes 2024-09-15T00:43:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3851770880 bytes 2024-09-15T00:43:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3908218880 bytes 2024-09-15T00:43:58.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3965609984 bytes 2024-09-15T00:43:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3980290048 bytes 2024-09-15T00:43:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 4037329920 bytes 2024-09-15T00:43:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 4052297728 bytes 2024-09-15T00:43:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 4081622016 bytes 2024-09-15T00:43:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 4089785344 bytes 2024-09-15T00:43:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:43:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:43:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:43:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-15T00:43:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:43:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:43:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:43:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:43:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:43:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:43:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:43:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:43:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:43:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:43:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:43:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:43:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:43:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:43:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+replaying 2024-09-15T00:43:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360961,"remote_snapshot_timestamp":1726360961,"replay_state":"idle"} 2024-09-15T00:43:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: service: mirror.0 on smithi059 2024-09-15T00:43:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:43:35 2024-09-15T00:43:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:43:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:43:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:43:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:43:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:43:52' 2024-09-15T00:43:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:43:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: state: up+replaying 2024-09-15T00:43:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ description: .*' 2024-09-15T00:43:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360961,"remote_snapshot_timestamp":1726360961,"replay_state":"idle"} 2024-09-15T00:43:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:43:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:43:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^up+' 2024-09-15T00:43:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:43:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep 'service: *mirror..* on ' 2024-09-15T00:43:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: service: mirror.0 on smithi059 2024-09-15T00:43:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update 2024-09-15T00:43:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-15T00:43:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:43:35' 2024-09-15T00:43:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-15 00:43:35' 2024-09-15T00:43:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:43:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:43:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local 'prev_last_update=2024-09-15 00:43:35' 2024-09-15T00:43:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:43:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:43:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-15T00:43:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='OKOKOK142224295mirror.3smithi05919.1.1-324-gecab7be5falseOK43564362mirror.2smithi05919.1.1-324-gecab7be5falseOK44794482mirror.0smithi05919.1.1-324-gecab7be5trueOK45464549mirror.1smithi05919.1.1-324-gecab7be5falseOKtest23953c4f-e4d9-4660-9569-50c96f4cf0e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726360961,"remote_snapshot_timestamp":1726360961,"replay_state":"idle"}44794482mirror.0smithi0592024-09-15 00:43:35cluster21108ad63-b7b4-4889-ae31-b1fd6de8431bup+stoppedlocal image is primary2024-09-15 00:43:52' 2024-09-15T00:43:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update state 2024-09-15T00:43:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-15T00:43:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:43:35' 2024-09-15T00:43:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-15T00:43:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=up+replaying 2024-09-15T00:43:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:43:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep up+replaying 2024-09-15T00:43:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:43:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:43:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap9 2024-09-15T00:43:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap9 2024-09-15T00:43:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:43:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:43:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:43:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap9 2024-09-15T00:43:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:44:28.017 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8982028054983353) > weight (0.25) 2024-09-15T00:44:28.017 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-15T00:44:28.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-15T00:44:28.018 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-15T00:44:28.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T00:44:28.017+0000 7f52d670f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 41871) UID: 1000 2024-09-15T00:44:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:44:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:44:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:44:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:44:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:44:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:44:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:44:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:44:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:44:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:44:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:44:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:44:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:44:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:44:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:44:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:44:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:44:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:44:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:44:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:44:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:44:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:44:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:44:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:44:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:44:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:44:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:44:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:44:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:44:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:44:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:44:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:44:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:44:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:44:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:44:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:44:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:44:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:44:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:44:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:44:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:44:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:44:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:44:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:44:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:44:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:44:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:44:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:44:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:44:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:44:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:44:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:44:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:44:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:44:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:44:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:44:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:44:33.019 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-15T00:44:33.019 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:44:33.020 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-15T00:44:33.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-15T00:44:33.020 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-15T00:44:33.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-15T00:44:33.023 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-15T00:44:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 29 2024-09-15T00:44:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:44:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:44:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:44:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:44:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:44:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:44:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=29 2024-09-15T00:44:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:44:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:44:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:44:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:44:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:44:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:44:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:44:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:44:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:44:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:44:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:44:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:44:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:44:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 29 = '' 2024-09-15T00:44:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:44:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:44:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:44:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:44:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:44:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:44:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:44:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:44:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:44:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:44:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:44:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:44:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 29 = '' 2024-09-15T00:44:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:44:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:44:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:44:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:44:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:44:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:44:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:44:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:44:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:44:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:44:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:44:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:44:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 29 = '' 2024-09-15T00:44:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:44:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:44:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:44:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:44:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:44:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:44:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=29 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 29 = 29 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap9 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:44:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:44:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-15T00:44:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:44:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:44:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:44:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 18 2024-09-15T00:44:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:44:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:43:44 2024 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:44:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:44:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:44:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snap8-clean ']' 2024-09-15T00:44:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-15T00:44:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap8-clean 2024-09-15T00:44:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:44:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:44:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:44:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:44:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:44:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:44:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:44:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:44:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:44:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:44:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:44:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap9 2024-09-15T00:44:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap9 2024-09-15T00:44:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:44:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:44:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:44:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:44:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:44:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:44:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:44:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:44:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:44:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:44:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name=snap9-clean 2024-09-15T00:44:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-15T00:44:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:44:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap9-clean 2024-09-15T00:44:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-15T00:44:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:44:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror test 2024-09-15T00:44:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:44:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:44:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=14 2024-09-15T00:44:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:44:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-15T00:44:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 14406656 bytes 2024-09-15T00:44:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 42148864 bytes 2024-09-15T00:44:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 56023040 bytes 2024-09-15T00:44:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 97821696 bytes 2024-09-15T00:44:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 98425856 bytes 2024-09-15T00:44:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 133521408 bytes 2024-09-15T00:44:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 134138880 bytes 2024-09-15T00:44:42.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 223904768 bytes 2024-09-15T00:44:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 275161088 bytes 2024-09-15T00:44:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 345216000 bytes 2024-09-15T00:44:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 409730048 bytes 2024-09-15T00:44:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 465092608 bytes 2024-09-15T00:44:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 566166528 bytes 2024-09-15T00:44:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 618544128 bytes 2024-09-15T00:44:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 681216000 bytes 2024-09-15T00:44:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 697286656 bytes 2024-09-15T00:44:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 721201152 bytes 2024-09-15T00:44:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 753903616 bytes 2024-09-15T00:44:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 800539648 bytes 2024-09-15T00:44:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 814456832 bytes 2024-09-15T00:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 857364480 bytes 2024-09-15T00:44:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 869741568 bytes 2024-09-15T00:44:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 902546432 bytes 2024-09-15T00:44:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 956725248 bytes 2024-09-15T00:44:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 965243904 bytes 2024-09-15T00:44:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 989940736 bytes 2024-09-15T00:44:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1000279040 bytes 2024-09-15T00:44:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1064476672 bytes 2024-09-15T00:44:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1095473152 bytes 2024-09-15T00:44:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1104460800 bytes 2024-09-15T00:44:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1153485824 bytes 2024-09-15T00:44:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1239586816 bytes 2024-09-15T00:44:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1250115584 bytes 2024-09-15T00:44:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1308749824 bytes 2024-09-15T00:44:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1364039680 bytes 2024-09-15T00:44:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1403419648 bytes 2024-09-15T00:44:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1436268544 bytes 2024-09-15T00:44:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1471855616 bytes 2024-09-15T00:44:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1516209152 bytes 2024-09-15T00:44:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1575243776 bytes 2024-09-15T00:44:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1614395392 bytes 2024-09-15T00:44:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1662410752 bytes 2024-09-15T00:44:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1673670656 bytes 2024-09-15T00:44:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1711787008 bytes 2024-09-15T00:44:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1778370560 bytes 2024-09-15T00:44:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1798724608 bytes 2024-09-15T00:44:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1807047680 bytes 2024-09-15T00:44:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1815202816 bytes 2024-09-15T00:44:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1838571520 bytes 2024-09-15T00:44:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1839204352 bytes 2024-09-15T00:44:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1857537024 bytes 2024-09-15T00:44:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 1900796928 bytes 2024-09-15T00:44:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 1901413376 bytes 2024-09-15T00:44:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 1935700992 bytes 2024-09-15T00:44:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 1945464832 bytes 2024-09-15T00:44:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 1946075136 bytes 2024-09-15T00:44:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 1959906304 bytes 2024-09-15T00:44:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 1993840640 bytes 2024-09-15T00:44:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 1996781568 bytes 2024-09-15T00:44:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 1999727616 bytes 2024-09-15T00:44:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2024087552 bytes 2024-09-15T00:44:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2071121920 bytes 2024-09-15T00:44:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2147275776 bytes 2024-09-15T00:44:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2185885696 bytes 2024-09-15T00:44:49.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2206574592 bytes 2024-09-15T00:44:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2232752128 bytes 2024-09-15T00:44:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2280223744 bytes 2024-09-15T00:44:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2302807040 bytes 2024-09-15T00:44:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2376562688 bytes 2024-09-15T00:44:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2416002048 bytes 2024-09-15T00:44:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2461488128 bytes 2024-09-15T00:44:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2467677184 bytes 2024-09-15T00:44:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2503348224 bytes 2024-09-15T00:44:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2527759360 bytes 2024-09-15T00:44:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2528726016 bytes 2024-09-15T00:44:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2535992320 bytes 2024-09-15T00:44:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 2597474304 bytes 2024-09-15T00:44:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 2630087680 bytes 2024-09-15T00:44:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 2662624256 bytes 2024-09-15T00:44:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 2708733952 bytes 2024-09-15T00:44:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 2783719424 bytes 2024-09-15T00:44:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 2831469568 bytes 2024-09-15T00:44:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 2916682752 bytes 2024-09-15T00:44:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 2946089984 bytes 2024-09-15T00:44:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3001350144 bytes 2024-09-15T00:44:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3026016256 bytes 2024-09-15T00:44:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3054686208 bytes 2024-09-15T00:44:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3091460096 bytes 2024-09-15T00:44:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3092626432 bytes 2024-09-15T00:44:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3111689216 bytes 2024-09-15T00:44:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3153893376 bytes 2024-09-15T00:44:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3216037888 bytes 2024-09-15T00:44:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3259496448 bytes 2024-09-15T00:44:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3336235008 bytes 2024-09-15T00:44:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 3392685056 bytes 2024-09-15T00:44:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 3420057600 bytes 2024-09-15T00:44:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 3458992128 bytes 2024-09-15T00:44:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 3519186944 bytes 2024-09-15T00:44:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 3554806784 bytes 2024-09-15T00:44:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 3581324288 bytes 2024-09-15T00:44:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 3602909184 bytes 2024-09-15T00:44:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 3637828608 bytes 2024-09-15T00:44:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 3670447104 bytes 2024-09-15T00:44:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 3721311232 bytes 2024-09-15T00:44:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 3745164288 bytes 2024-09-15T00:44:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 3768933376 bytes 2024-09-15T00:44:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 3797369856 bytes 2024-09-15T00:44:54.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 3853019136 bytes 2024-09-15T00:44:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 3979721728 bytes 2024-09-15T00:44:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 4026905600 bytes 2024-09-15T00:44:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11100 IOs, 4070304768 bytes 2024-09-15T00:44:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11200 IOs, 4100105216 bytes 2024-09-15T00:44:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11300 IOs, 4148551680 bytes 2024-09-15T00:44:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11400 IOs, 4159846400 bytes 2024-09-15T00:44:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11500 IOs, 4212261888 bytes 2024-09-15T00:44:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11600 IOs, 4242407424 bytes 2024-09-15T00:44:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11700 IOs, 4242993152 bytes 2024-09-15T00:44:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11800 IOs, 4331120640 bytes 2024-09-15T00:44:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11900 IOs, 4339982336 bytes 2024-09-15T00:44:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12000 IOs, 4413019136 bytes 2024-09-15T00:44:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:44:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-15T00:44:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:44:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local description_pattern= 2024-09-15T00:44:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local service_pattern= 2024-09-15T00:44:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:44:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:44:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-15T00:44:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='test: 2024-09-15T00:44:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: global_id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:44:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+replaying 2024-09-15T00:44:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534682624.0,"last_snapshot_bytes":534682624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726361018,"remote_snapshot_timestamp":1726361018,"replay_state":"idle"} 2024-09-15T00:44:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: service: mirror.0 on smithi059 2024-09-15T00:44:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:44:34 2024-09-15T00:44:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: peer_sites: 2024-09-15T00:44:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: name: cluster2 2024-09-15T00:44:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: state: up+stopped 2024-09-15T00:44:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: description: local image is primary 2024-09-15T00:44:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: last_update: 2024-09-15 00:44:52' 2024-09-15T00:44:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ state: .*up+replaying' 2024-09-15T00:44:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: state: up+replaying 2024-09-15T00:44:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^ description: .*' 2024-09-15T00:44:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534682624.0,"last_snapshot_bytes":534682624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726361018,"remote_snapshot_timestamp":1726361018,"replay_state":"idle"} 2024-09-15T00:44:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:44:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:44:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep '^up+' 2024-09-15T00:44:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:44:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep 'service: *mirror..* on ' 2024-09-15T00:44:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: service: mirror.0 on smithi059 2024-09-15T00:44:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update 2024-09-15T00:44:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-15T00:44:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:44:34' 2024-09-15T00:44:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-15 00:44:34' 2024-09-15T00:44:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:44:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state_pattern=up+replaying 2024-09-15T00:44:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local 'prev_last_update=2024-09-15 00:44:34' 2024-09-15T00:44:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status 2024-09-15T00:44:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ CEPH_ARGS= 2024-09-15T00:44:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-15T00:44:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status='OKOKOK142224295mirror.3smithi05919.1.1-324-gecab7be5falseOK43564362mirror.2smithi05919.1.1-324-gecab7be5falseOK44794482mirror.0smithi05919.1.1-324-gecab7be5trueOK45464549mirror.1smithi05919.1.1-324-gecab7be5falseOKtest23953c4f-e4d9-4660-9569-50c96f4cf0e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534682624.0,"last_snapshot_bytes":534682624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726361018,"remote_snapshot_timestamp":1726361018,"replay_state":"idle"}44794482mirror.0smithi0592024-09-15 00:44:34cluster21108ad63-b7b4-4889-ae31-b1fd6de8431bup+stoppedlocal image is primary2024-09-15 00:44:52' 2024-09-15T00:44:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_update state 2024-09-15T00:44:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-15T00:44:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ last_update='2024-09-15 00:44:34' 2024-09-15T00:44:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-15T00:44:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=up+replaying 2024-09-15T00:44:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo up+replaying 2024-09-15T00:44:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep up+replaying 2024-09-15T00:44:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:up+replaying 2024-09-15T00:44:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:44:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:44:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap10 2024-09-15T00:44:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap10 2024-09-15T00:44:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:44:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:44:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:44:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap10 2024-09-15T00:44:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:45:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:45:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:45:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:45:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:45:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:45:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:45:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:45:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:45:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:45:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:45:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:45:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:45:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:45:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:45:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:45:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:45:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:45:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:45:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:45:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:45:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:45:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:45:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:45:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:45:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:45:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:45:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:45:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:45:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:45:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:45:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:45:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:45:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:45:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:45:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:45:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:45:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:45:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:45:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:45:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:45:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:45:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:45:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:45:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:45:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:45:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:45:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:45:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:45:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:45:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:45:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:45:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:45:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:45:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:45:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:45:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:45:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:45:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:45:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:45:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:45:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:45:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:45:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:45:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:45:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:45:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:45:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:45:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:45:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:45:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:45:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:45:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:45:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:45:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:45:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 32 2024-09-15T00:45:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:45:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:45:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:45:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:45:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:45:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:45:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=32 2024-09-15T00:45:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:45:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:45:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:45:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:45:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:45:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:45:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:45:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:45:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:45:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:45:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:45:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 32 = '' 2024-09-15T00:45:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:45:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:45:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:45:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:45:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:45:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:45:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:45:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:45:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:45:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:45:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 32 = '' 2024-09-15T00:45:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:45:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:45:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:45:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:45:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:45:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:45:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:45:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:45:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:45:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 32 = '' 2024-09-15T00:45:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:45:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:45:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:45:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:45:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:45:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=32 2024-09-15T00:45:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 32 = 32 2024-09-15T00:45:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-15T00:45:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap10 2024-09-15T00:45:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:45:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:45:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:45:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-15T00:45:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:45:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 20 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:43:44 2024 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:45:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:45:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:45:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:45:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:45:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snap9-clean ']' 2024-09-15T00:45:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-15T00:45:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap9-clean 2024-09-15T00:45:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:33.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap10 2024-09-15T00:45:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap10 2024-09-15T00:45:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ clean_snap_name=snap10-clean 2024-09-15T00:45:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-15T00:45:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:45:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap10-clean 2024-09-15T00:45:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-15T00:45:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-15T00:45:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-15T00:45:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:45:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:45:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:45:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:45:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-15T00:45:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:45:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:45:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:45:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:45:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-15T00:45:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:45:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:45:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:45:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:45:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:45:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:45:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:45:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:45:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:45:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:45:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:45:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:45:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:45:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:45:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:45:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:45:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:45:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:45:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:45:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:45:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:45:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:45:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:45:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:45:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:45:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:45:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:45:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:45:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:45:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:45:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:45:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:45:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:45:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-15T00:45:37.814 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7445380112001055) > weight (0.25) 2024-09-15T00:45:37.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3159511680677085) > weight (0.25) 2024-09-15T00:45:37.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.607985474247525) > weight (0.25) 2024-09-15T00:45:37.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3253959134406659) > weight (0.25) 2024-09-15T00:45:37.815 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-09-15T00:45:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:45:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:45:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:45:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:45:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:45:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:45:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:45:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-15T00:45:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:45:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:45:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:45:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:45:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-15T00:45:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:45:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:45:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-test.status 2024-09-15T00:45:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-test.status 2024-09-15T00:45:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:45:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:45:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-test.status 2024-09-15T00:45:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-test.status 2024-09-15T00:45:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:45:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-15T00:45:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:45:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-15T00:45:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 34 2024-09-15T00:45:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:45:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:45:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.status 2024-09-15T00:45:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-15T00:45:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:45:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:45:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=34 2024-09-15T00:45:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:45:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:45:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:45:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:45:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:45:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:45:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:45:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:45:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:45:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:45:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:45:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 34 = '' 2024-09-15T00:45:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:45:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:45:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.status 2024-09-15T00:45:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-15T00:45:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:45:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:45:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=34 2024-09-15T00:45:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 34 = 34 2024-09-15T00:45:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-15T00:45:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:45:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap10-clean 2024-09-15T00:45:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:45:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:45:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:45:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-15T00:45:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'test': 2024-09-15T00:45:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T00:45:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:45:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 21 2024-09-15T00:45:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 10a8272a6f72 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.10a8272a6f72 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:45:33 2024 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:36:41 2024 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:45:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23953c4f-e4d9-4660-9569-50c96f4cf0e9 2024-09-15T00:45:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:45:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 10 2024-09-15T00:45:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:45:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap1 2024-09-15T00:45:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap1 2024-09-15T00:45:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap1 2024-09-15T00:45:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap1-clean 2024-09-15T00:45:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-15T00:45:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap1-clean 2024-09-15T00:45:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:43.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:45:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap2 2024-09-15T00:45:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap2 2024-09-15T00:45:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap2 2024-09-15T00:45:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap2-clean 2024-09-15T00:45:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-15T00:45:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap2-clean 2024-09-15T00:45:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:45:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap3 2024-09-15T00:45:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap3 2024-09-15T00:45:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap3 2024-09-15T00:45:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:48.024 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-15T00:45:48.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-15T00:45:48.025 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-15T00:45:48.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T00:45:48.024+0000 7f907036f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 36703) UID: 1000 2024-09-15T00:45:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap3-clean 2024-09-15T00:45:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-15T00:45:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap3-clean 2024-09-15T00:45:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:45:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap4 2024-09-15T00:45:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap4 2024-09-15T00:45:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap4 2024-09-15T00:45:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap4-clean 2024-09-15T00:45:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-15T00:45:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap4-clean 2024-09-15T00:45:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:45:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap5 2024-09-15T00:45:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap5 2024-09-15T00:45:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap5 2024-09-15T00:45:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap5-clean 2024-09-15T00:45:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-15T00:45:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap5-clean 2024-09-15T00:45:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:57.026 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-15T00:45:57.026 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:45:57.027 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-15T00:45:57.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-15T00:45:57.027 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-15T00:45:57.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-15T00:45:57.030 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-09-15T00:45:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:45:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap6 2024-09-15T00:45:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap6 2024-09-15T00:45:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap6 2024-09-15T00:45:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:45:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap6-clean 2024-09-15T00:45:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-15T00:45:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:45:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:45:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap6-clean 2024-09-15T00:45:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:45:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:45:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:45:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:45:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap7 2024-09-15T00:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap7 2024-09-15T00:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap7 2024-09-15T00:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap7-clean 2024-09-15T00:46:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-15T00:46:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap7-clean 2024-09-15T00:46:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:46:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap8 2024-09-15T00:46:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap8 2024-09-15T00:46:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap8 2024-09-15T00:46:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:46:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap8-clean 2024-09-15T00:46:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-15T00:46:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap8-clean 2024-09-15T00:46:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:46:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap9 2024-09-15T00:46:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap9 2024-09-15T00:46:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap9 2024-09-15T00:46:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:46:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap9-clean 2024-09-15T00:46:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-15T00:46:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap9-clean 2024-09-15T00:46:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:46:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap10 2024-09-15T00:46:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap10 2024-09-15T00:46:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap10 2024-09-15T00:46:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:46:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap10-clean 2024-09-15T00:46:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-15T00:46:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap10-clean 2024-09-15T00:46:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:46:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export 2024-09-15T00:46:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-15T00:46:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-test.export /tmp/tmp.nS85ynkalX/cluster1-mirror-test.export 2024-09-15T00:46:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 10 2024-09-15T00:46:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap1 2024-09-15T00:46:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-15T00:46:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap1 2024-09-15T00:46:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-15T00:46:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap1-clean 2024-09-15T00:46:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-15T00:46:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap1-clean 2024-09-15T00:46:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-15T00:46:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap2 2024-09-15T00:46:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-15T00:46:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap2 2024-09-15T00:46:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-15T00:46:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap2-clean 2024-09-15T00:46:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-15T00:46:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap2-clean 2024-09-15T00:46:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-15T00:46:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap3 2024-09-15T00:46:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-15T00:46:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap3 2024-09-15T00:46:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-15T00:46:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap3-clean 2024-09-15T00:46:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-15T00:46:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap3-clean 2024-09-15T00:46:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-15T00:46:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap4 2024-09-15T00:46:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-15T00:46:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap4 2024-09-15T00:46:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-15T00:46:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap4-clean 2024-09-15T00:46:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-15T00:46:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap4-clean 2024-09-15T00:46:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-15T00:46:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap5 2024-09-15T00:46:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-15T00:46:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap5 2024-09-15T00:46:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-15T00:46:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap5-clean 2024-09-15T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-15T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap5-clean 2024-09-15T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-15T00:46:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap6 2024-09-15T00:46:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-15T00:46:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap6 2024-09-15T00:46:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-15T00:46:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap6-clean 2024-09-15T00:46:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-15T00:46:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap6-clean 2024-09-15T00:46:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-15T00:46:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap7 2024-09-15T00:46:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-15T00:46:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap7 2024-09-15T00:46:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-15T00:46:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap7-clean 2024-09-15T00:46:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-15T00:46:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap7-clean 2024-09-15T00:46:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-15T00:46:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap8 2024-09-15T00:46:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-15T00:46:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap8 2024-09-15T00:46:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-15T00:46:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap8-clean 2024-09-15T00:46:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-15T00:46:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap8-clean 2024-09-15T00:46:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-15T00:46:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap9 2024-09-15T00:46:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-15T00:46:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap9 2024-09-15T00:46:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-15T00:46:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap9-clean 2024-09-15T00:46:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-15T00:46:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap9-clean 2024-09-15T00:46:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-15T00:46:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T00:46:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap10 2024-09-15T00:46:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-15T00:46:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap10 2024-09-15T00:46:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-15T00:46:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap10-clean 2024-09-15T00:46:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-15T00:46:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap10-clean 2024-09-15T00:46:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-15T00:46:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T00:46:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror test 2024-09-15T00:46:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T00:46:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T00:46:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror test 2024-09-15T00:46:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-15T00:46:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-15T00:46:34.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T00:46:33.998+0000 7f84be7c0640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560f1534b860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T00:46:34.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T00:46:33.998+0000 7f84c4fcd640 -1 rbd::mirror::ImageReplayer: 0x560f15c64000 [3/23953c4f-e4d9-4660-9569-50c96f4cf0e9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T00:46:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T00:46:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-15T00:46:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:46:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T00:46:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T00:46:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:46:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T00:46:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror test 2024-09-15T00:46:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:46:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T00:46:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=test 2024-09-15T00:46:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-15T00:46:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T00:46:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-15T00:46:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T00:46:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T00:46:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T00:46:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-15T00:46:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:46:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=test 2024-09-15T00:46:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T00:46:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T00:46:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T00:46:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T00:46:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror test 2024-09-15T00:46:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:46:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T00:46:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=test 2024-09-15T00:46:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-15T00:46:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T00:46:36.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-15T00:46:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T00:46:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T00:46:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T00:46:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ testlog 'TEST: create many images' 2024-09-15T00:46:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ tee -a /tmp/tmp.nS85ynkalX/rbd-mirror.test.log 2024-09-15T00:46:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ date '+%F %T' 2024-09-15T00:46:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo 2024-09-15 00:46:36 TEST: create many images 2024-09-15T00:46:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:2024-09-15 00:46:36 TEST: create many images 2024-09-15T00:46:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ snap_name=snap 2024-09-15T00:46:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 50 2024-09-15T00:46:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:46:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_1 2024-09-15T00:46:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-15T00:46:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:46:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:46:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:46:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-15T00:46:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-15T00:46:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:46:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:46:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-15T00:46:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:46:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:46:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-15T00:46:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:46:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-15T00:46:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_1': 2024-09-15T00:46:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:46:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:46:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:46:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1265a76f4250 2024-09-15T00:46:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1265a76f4250 2024-09-15T00:46:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:46:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:46:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:46:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:46:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:46:36 2024 2024-09-15T00:46:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:46:36 2024 2024-09-15T00:46:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:46:36 2024 2024-09-15T00:46:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:46:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:46:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: f6dc2e78-22b9-4458-a5da-18ecc500097e 2024-09-15T00:46:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-15T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=14 2024-09-15T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:46:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:46:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 21226496 bytes 2024-09-15T00:46:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 22719488 bytes 2024-09-15T00:46:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 48876544 bytes 2024-09-15T00:46:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 70857728 bytes 2024-09-15T00:46:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 125100032 bytes 2024-09-15T00:46:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 125714432 bytes 2024-09-15T00:46:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 157345792 bytes 2024-09-15T00:46:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 176908288 bytes 2024-09-15T00:46:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 227504128 bytes 2024-09-15T00:46:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 257124352 bytes 2024-09-15T00:46:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 280958976 bytes 2024-09-15T00:46:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 327985152 bytes 2024-09-15T00:46:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 356101120 bytes 2024-09-15T00:46:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 400482304 bytes 2024-09-15T00:46:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 443261952 bytes 2024-09-15T00:46:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 489273344 bytes 2024-09-15T00:46:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 506362880 bytes 2024-09-15T00:46:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 524413952 bytes 2024-09-15T00:46:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 547634176 bytes 2024-09-15T00:46:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 614463488 bytes 2024-09-15T00:46:38.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 633149440 bytes 2024-09-15T00:46:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 682368000 bytes 2024-09-15T00:46:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 711251968 bytes 2024-09-15T00:46:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 716363776 bytes 2024-09-15T00:46:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 753466368 bytes 2024-09-15T00:46:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 792029184 bytes 2024-09-15T00:46:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 837397504 bytes 2024-09-15T00:46:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 911278080 bytes 2024-09-15T00:46:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 937140224 bytes 2024-09-15T00:46:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 952435712 bytes 2024-09-15T00:46:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1014113280 bytes 2024-09-15T00:46:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1024051200 bytes 2024-09-15T00:46:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1070166016 bytes 2024-09-15T00:46:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1122956288 bytes 2024-09-15T00:46:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1237774336 bytes 2024-09-15T00:46:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1270685696 bytes 2024-09-15T00:46:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1351510016 bytes 2024-09-15T00:46:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1382645760 bytes 2024-09-15T00:46:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1397309440 bytes 2024-09-15T00:46:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1444635648 bytes 2024-09-15T00:46:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1517572096 bytes 2024-09-15T00:46:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1534600192 bytes 2024-09-15T00:46:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1606926336 bytes 2024-09-15T00:46:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1650865152 bytes 2024-09-15T00:46:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1689082880 bytes 2024-09-15T00:46:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1799200768 bytes 2024-09-15T00:46:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1846588416 bytes 2024-09-15T00:46:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1861892096 bytes 2024-09-15T00:46:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1908102144 bytes 2024-09-15T00:46:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1923827712 bytes 2024-09-15T00:46:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1958029312 bytes 2024-09-15T00:46:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2027230208 bytes 2024-09-15T00:46:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2064999424 bytes 2024-09-15T00:46:42.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2153141248 bytes 2024-09-15T00:46:42.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2205720576 bytes 2024-09-15T00:46:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2230874112 bytes 2024-09-15T00:46:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2257022976 bytes 2024-09-15T00:46:42.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2257593344 bytes 2024-09-15T00:46:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2308981760 bytes 2024-09-15T00:46:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2310114304 bytes 2024-09-15T00:46:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2358516736 bytes 2024-09-15T00:46:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2430314496 bytes 2024-09-15T00:46:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2509455360 bytes 2024-09-15T00:46:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2543959040 bytes 2024-09-15T00:46:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2617233408 bytes 2024-09-15T00:46:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2662541312 bytes 2024-09-15T00:46:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2694666240 bytes 2024-09-15T00:46:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2711468032 bytes 2024-09-15T00:46:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2803450880 bytes 2024-09-15T00:46:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2876925952 bytes 2024-09-15T00:46:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2933955584 bytes 2024-09-15T00:46:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2966268928 bytes 2024-09-15T00:46:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2994498560 bytes 2024-09-15T00:46:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3069270016 bytes 2024-09-15T00:46:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3129604096 bytes 2024-09-15T00:46:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3287949312 bytes 2024-09-15T00:46:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3298168832 bytes 2024-09-15T00:46:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3348072448 bytes 2024-09-15T00:46:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3360868352 bytes 2024-09-15T00:46:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3399161856 bytes 2024-09-15T00:46:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3454973952 bytes 2024-09-15T00:46:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3512143872 bytes 2024-09-15T00:46:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3572799488 bytes 2024-09-15T00:46:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3594339328 bytes 2024-09-15T00:46:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3644898304 bytes 2024-09-15T00:46:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3730397184 bytes 2024-09-15T00:46:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3768512512 bytes 2024-09-15T00:46:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3830209536 bytes 2024-09-15T00:46:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3897857024 bytes 2024-09-15T00:46:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3905615872 bytes 2024-09-15T00:46:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 4022938624 bytes 2024-09-15T00:46:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 4109576192 bytes 2024-09-15T00:46:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 4196421632 bytes 2024-09-15T00:46:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4215365632 bytes 2024-09-15T00:46:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4282020864 bytes 2024-09-15T00:46:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4321677312 bytes 2024-09-15T00:46:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4376483840 bytes 2024-09-15T00:46:47.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4394749952 bytes 2024-09-15T00:46:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4426502144 bytes 2024-09-15T00:46:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4469115904 bytes 2024-09-15T00:46:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4492451840 bytes 2024-09-15T00:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4598798336 bytes 2024-09-15T00:46:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4599433216 bytes 2024-09-15T00:46:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4635789312 bytes 2024-09-15T00:46:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4711162880 bytes 2024-09-15T00:46:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 4793214976 bytes 2024-09-15T00:46:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4817276928 bytes 2024-09-15T00:46:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 4855501824 bytes 2024-09-15T00:46:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 4934759424 bytes 2024-09-15T00:46:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 4960548864 bytes 2024-09-15T00:46:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11100 IOs, 5003848704 bytes 2024-09-15T00:46:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11200 IOs, 5020519424 bytes 2024-09-15T00:46:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11300 IOs, 5074415616 bytes 2024-09-15T00:46:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11400 IOs, 5083253760 bytes 2024-09-15T00:46:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11500 IOs, 5107428352 bytes 2024-09-15T00:46:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11600 IOs, 5243141120 bytes 2024-09-15T00:46:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11700 IOs, 5266374656 bytes 2024-09-15T00:46:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11800 IOs, 5289526272 bytes 2024-09-15T00:46:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11900 IOs, 5337039872 bytes 2024-09-15T00:46:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12000 IOs, 5374530560 bytes 2024-09-15T00:46:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12100 IOs, 5415925760 bytes 2024-09-15T00:46:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12200 IOs, 5435173888 bytes 2024-09-15T00:46:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12300 IOs, 5465649152 bytes 2024-09-15T00:46:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_2 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:46:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:46:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:46:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:46:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-15T00:46:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:46:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:46:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:46:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-15T00:46:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:46:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:46:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-15T00:46:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:46:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-15T00:46:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_2': 2024-09-15T00:46:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:46:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:46:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:46:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1271bfbb42 2024-09-15T00:46:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1271bfbb42 2024-09-15T00:46:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:46:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:46:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:46:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:46:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:46:51 2024 2024-09-15T00:46:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:46:51 2024 2024-09-15T00:46:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:46:51 2024 2024-09-15T00:46:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:46:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:46:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 52f7a6a8-1ded-4b8e-b848-e1f2de6a62b1 2024-09-15T00:46:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:46:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:46:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-15T00:46:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:46:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:46:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:46:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:46:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=11 2024-09-15T00:46:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:46:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:46:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 32004096 bytes 2024-09-15T00:46:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 47837184 bytes 2024-09-15T00:46:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 48440320 bytes 2024-09-15T00:46:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 54688768 bytes 2024-09-15T00:46:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 55320576 bytes 2024-09-15T00:46:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 59359232 bytes 2024-09-15T00:46:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 118807552 bytes 2024-09-15T00:46:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 175714304 bytes 2024-09-15T00:46:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 205553664 bytes 2024-09-15T00:46:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 225714176 bytes 2024-09-15T00:46:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 297510912 bytes 2024-09-15T00:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 319644672 bytes 2024-09-15T00:46:53.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 371091456 bytes 2024-09-15T00:46:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 385196032 bytes 2024-09-15T00:46:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 409937920 bytes 2024-09-15T00:46:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 472030208 bytes 2024-09-15T00:46:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 543803392 bytes 2024-09-15T00:46:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 653067264 bytes 2024-09-15T00:46:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 700622848 bytes 2024-09-15T00:46:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 765732864 bytes 2024-09-15T00:46:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 783392768 bytes 2024-09-15T00:46:54.022 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T00:46:54.019+0000 7fc075401640 -1 reset not still connected to 0x55a0bb318410 2024-09-15T00:46:54.022 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T00:46:54.019+0000 7fc075401640 -1 reset not still connected to 0x55a0bb318750 2024-09-15T00:46:54.022 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T00:46:54.019+0000 7fc075401640 -1 reset not still connected to 0x55a0bdc0de10 2024-09-15T00:46:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 850485248 bytes 2024-09-15T00:46:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 938645504 bytes 2024-09-15T00:46:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1034030080 bytes 2024-09-15T00:46:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1094832128 bytes 2024-09-15T00:46:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1143877632 bytes 2024-09-15T00:46:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1206323200 bytes 2024-09-15T00:46:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1240204288 bytes 2024-09-15T00:46:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1259805696 bytes 2024-09-15T00:46:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1269799936 bytes 2024-09-15T00:46:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1332114432 bytes 2024-09-15T00:46:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1393864704 bytes 2024-09-15T00:46:55.482 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1435617280 bytes 2024-09-15T00:46:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1516411904 bytes 2024-09-15T00:46:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1663206400 bytes 2024-09-15T00:46:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1709500416 bytes 2024-09-15T00:46:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1741002752 bytes 2024-09-15T00:46:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1801676800 bytes 2024-09-15T00:46:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1844593664 bytes 2024-09-15T00:46:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1869291520 bytes 2024-09-15T00:46:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1907654656 bytes 2024-09-15T00:46:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1915049984 bytes 2024-09-15T00:46:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1938897920 bytes 2024-09-15T00:46:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1999967232 bytes 2024-09-15T00:46:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2019276800 bytes 2024-09-15T00:46:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2051205120 bytes 2024-09-15T00:46:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2074129408 bytes 2024-09-15T00:46:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2110661632 bytes 2024-09-15T00:46:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2159440896 bytes 2024-09-15T00:46:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2210409472 bytes 2024-09-15T00:46:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2233112576 bytes 2024-09-15T00:46:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2233733120 bytes 2024-09-15T00:46:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2324364288 bytes 2024-09-15T00:46:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2365346816 bytes 2024-09-15T00:46:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2365969408 bytes 2024-09-15T00:46:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2387548160 bytes 2024-09-15T00:46:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2440459264 bytes 2024-09-15T00:46:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2477258752 bytes 2024-09-15T00:46:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2495903744 bytes 2024-09-15T00:46:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2498620416 bytes 2024-09-15T00:46:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2559616000 bytes 2024-09-15T00:46:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2603819008 bytes 2024-09-15T00:46:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2632573952 bytes 2024-09-15T00:46:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2681317376 bytes 2024-09-15T00:46:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2696713216 bytes 2024-09-15T00:46:59.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2715459584 bytes 2024-09-15T00:46:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2735132672 bytes 2024-09-15T00:46:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2767643648 bytes 2024-09-15T00:46:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2847295488 bytes 2024-09-15T00:46:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2858500096 bytes 2024-09-15T00:46:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2883922944 bytes 2024-09-15T00:46:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2974972928 bytes 2024-09-15T00:47:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3087077376 bytes 2024-09-15T00:47:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3126115328 bytes 2024-09-15T00:47:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3167968256 bytes 2024-09-15T00:47:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3233476608 bytes 2024-09-15T00:47:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3257888768 bytes 2024-09-15T00:47:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3348803584 bytes 2024-09-15T00:47:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3428436992 bytes 2024-09-15T00:47:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3448562688 bytes 2024-09-15T00:47:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3483132928 bytes 2024-09-15T00:47:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3567579136 bytes 2024-09-15T00:47:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3590519808 bytes 2024-09-15T00:47:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3630000128 bytes 2024-09-15T00:47:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3659038720 bytes 2024-09-15T00:47:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3687056384 bytes 2024-09-15T00:47:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3745713152 bytes 2024-09-15T00:47:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3749559296 bytes 2024-09-15T00:47:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3775489024 bytes 2024-09-15T00:47:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3823380480 bytes 2024-09-15T00:47:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3865300992 bytes 2024-09-15T00:47:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3888602112 bytes 2024-09-15T00:47:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3933894656 bytes 2024-09-15T00:47:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4017730560 bytes 2024-09-15T00:47:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4035616768 bytes 2024-09-15T00:47:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4069634048 bytes 2024-09-15T00:47:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4105816064 bytes 2024-09-15T00:47:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:47:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:47:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:47:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:47:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:47:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_3 2024-09-15T00:47:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-15T00:47:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:47:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:47:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:47:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:47:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-15T00:47:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:47:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:47:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:47:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-15T00:47:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:47:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:47:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-15T00:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:47:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-15T00:47:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_3': 2024-09-15T00:47:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:47:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:47:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:47:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 127da1eb2fab 2024-09-15T00:47:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.127da1eb2fab 2024-09-15T00:47:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:47:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:47:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:47:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:47:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:47:02 2024 2024-09-15T00:47:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:47:02 2024 2024-09-15T00:47:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:47:02 2024 2024-09-15T00:47:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:47:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:47:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 3262a16a-009b-44a2-b91f-62ecc486851b 2024-09-15T00:47:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:47:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:47:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-15T00:47:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:47:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:47:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=7 2024-09-15T00:47:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:47:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:47:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 24163328 bytes 2024-09-15T00:47:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 132312064 bytes 2024-09-15T00:47:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 213347328 bytes 2024-09-15T00:47:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 275461120 bytes 2024-09-15T00:47:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 302453760 bytes 2024-09-15T00:47:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 382642176 bytes 2024-09-15T00:47:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 444878848 bytes 2024-09-15T00:47:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 466429952 bytes 2024-09-15T00:47:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 507543552 bytes 2024-09-15T00:47:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 549040128 bytes 2024-09-15T00:47:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 575592448 bytes 2024-09-15T00:47:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 635959296 bytes 2024-09-15T00:47:05.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 701140992 bytes 2024-09-15T00:47:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 745279488 bytes 2024-09-15T00:47:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 783187968 bytes 2024-09-15T00:47:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 809313280 bytes 2024-09-15T00:47:05.816 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9995940524146593) > weight (0.25) 2024-09-15T00:47:05.816 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6497558403560163) > weight (0.25) 2024-09-15T00:47:05.816 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-15T00:47:05.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-15T00:47:05.817 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-15T00:47:05.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T00:47:05.816+0000 7ff6a2aea640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 39234) UID: 1000 2024-09-15T00:47:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 850652160 bytes 2024-09-15T00:47:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 901438464 bytes 2024-09-15T00:47:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 964560896 bytes 2024-09-15T00:47:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1045578752 bytes 2024-09-15T00:47:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1058428928 bytes 2024-09-15T00:47:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1113506816 bytes 2024-09-15T00:47:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1157555200 bytes 2024-09-15T00:47:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1274954752 bytes 2024-09-15T00:47:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1327669248 bytes 2024-09-15T00:47:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1386031104 bytes 2024-09-15T00:47:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1398686720 bytes 2024-09-15T00:47:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1420977152 bytes 2024-09-15T00:47:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1483511808 bytes 2024-09-15T00:47:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1532463104 bytes 2024-09-15T00:47:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1577660416 bytes 2024-09-15T00:47:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1627288576 bytes 2024-09-15T00:47:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1700061184 bytes 2024-09-15T00:47:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1779731456 bytes 2024-09-15T00:47:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1780344832 bytes 2024-09-15T00:47:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1793032192 bytes 2024-09-15T00:47:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1811299328 bytes 2024-09-15T00:47:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1887076352 bytes 2024-09-15T00:47:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1919622144 bytes 2024-09-15T00:47:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1944620032 bytes 2024-09-15T00:47:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1972374528 bytes 2024-09-15T00:47:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 2035916800 bytes 2024-09-15T00:47:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2085513216 bytes 2024-09-15T00:47:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2135322624 bytes 2024-09-15T00:47:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2135908352 bytes 2024-09-15T00:47:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2170335232 bytes 2024-09-15T00:47:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2207781888 bytes 2024-09-15T00:47:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2291654656 bytes 2024-09-15T00:47:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2354272256 bytes 2024-09-15T00:47:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2377741312 bytes 2024-09-15T00:47:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2417087488 bytes 2024-09-15T00:47:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2460743680 bytes 2024-09-15T00:47:09.818 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-15T00:47:09.818 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:47:09.819 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-15T00:47:09.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-15T00:47:09.819 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-15T00:47:09.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-15T00:47:09.822 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-15T00:47:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2500746240 bytes 2024-09-15T00:47:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2533381120 bytes 2024-09-15T00:47:10.031 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8037076532303615) > weight (0.25) 2024-09-15T00:47:10.031 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9324532021881596) > weight (0.25) 2024-09-15T00:47:10.031 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-15T00:47:10.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-15T00:47:10.032 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-15T00:47:10.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T00:47:10.031+0000 7f426a377640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 38939) UID: 1000 2024-09-15T00:47:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2535529472 bytes 2024-09-15T00:47:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2637536256 bytes 2024-09-15T00:47:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2685755392 bytes 2024-09-15T00:47:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2728367104 bytes 2024-09-15T00:47:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2743151616 bytes 2024-09-15T00:47:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2772856832 bytes 2024-09-15T00:47:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_4 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:47:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:47:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-15T00:47:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:47:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:47:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:47:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-15T00:47:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:47:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:47:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-15T00:47:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:47:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-15T00:47:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_4': 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 128eca798e6f 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.128eca798e6f 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:47:10 2024 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:47:10 2024 2024-09-15T00:47:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:47:10 2024 2024-09-15T00:47:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:47:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:47:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 51352989-1223-414e-86a9-443c3c234183 2024-09-15T00:47:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:47:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:47:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-15T00:47:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:47:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:47:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=14 2024-09-15T00:47:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:47:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:47:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 27321344 bytes 2024-09-15T00:47:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 67600384 bytes 2024-09-15T00:47:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 95524864 bytes 2024-09-15T00:47:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 129547264 bytes 2024-09-15T00:47:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 148386816 bytes 2024-09-15T00:47:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 170786816 bytes 2024-09-15T00:47:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 206435328 bytes 2024-09-15T00:47:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 259775488 bytes 2024-09-15T00:47:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 280302592 bytes 2024-09-15T00:47:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 398193664 bytes 2024-09-15T00:47:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 416292864 bytes 2024-09-15T00:47:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 428241920 bytes 2024-09-15T00:47:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 442848256 bytes 2024-09-15T00:47:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 527031296 bytes 2024-09-15T00:47:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 548830208 bytes 2024-09-15T00:47:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 629606400 bytes 2024-09-15T00:47:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 691714048 bytes 2024-09-15T00:47:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 761464832 bytes 2024-09-15T00:47:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 803131392 bytes 2024-09-15T00:47:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 829621248 bytes 2024-09-15T00:47:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 891335680 bytes 2024-09-15T00:47:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 922460160 bytes 2024-09-15T00:47:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 941617152 bytes 2024-09-15T00:47:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 979706880 bytes 2024-09-15T00:47:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 996001792 bytes 2024-09-15T00:47:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1029723136 bytes 2024-09-15T00:47:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1045662720 bytes 2024-09-15T00:47:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1046268928 bytes 2024-09-15T00:47:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1049671680 bytes 2024-09-15T00:47:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1103002624 bytes 2024-09-15T00:47:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1157051392 bytes 2024-09-15T00:47:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1213084672 bytes 2024-09-15T00:47:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1234775040 bytes 2024-09-15T00:47:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1270376448 bytes 2024-09-15T00:47:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1294088192 bytes 2024-09-15T00:47:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1305361408 bytes 2024-09-15T00:47:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1332206592 bytes 2024-09-15T00:47:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1344577536 bytes 2024-09-15T00:47:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1350226944 bytes 2024-09-15T00:47:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1435102208 bytes 2024-09-15T00:47:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1461783552 bytes 2024-09-15T00:47:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1501786112 bytes 2024-09-15T00:47:16.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1568799744 bytes 2024-09-15T00:47:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1634851840 bytes 2024-09-15T00:47:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1680280576 bytes 2024-09-15T00:47:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1730134016 bytes 2024-09-15T00:47:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1832366080 bytes 2024-09-15T00:47:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1936236544 bytes 2024-09-15T00:47:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2035815424 bytes 2024-09-15T00:47:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2040352768 bytes 2024-09-15T00:47:17.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2079168512 bytes 2024-09-15T00:47:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2093720576 bytes 2024-09-15T00:47:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2174373888 bytes 2024-09-15T00:47:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2226597888 bytes 2024-09-15T00:47:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2247225344 bytes 2024-09-15T00:47:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2263217152 bytes 2024-09-15T00:47:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2277249024 bytes 2024-09-15T00:47:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2397718528 bytes 2024-09-15T00:47:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2454874112 bytes 2024-09-15T00:47:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2528357376 bytes 2024-09-15T00:47:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2598256640 bytes 2024-09-15T00:47:19.033 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-15T00:47:19.033 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:47:19.033 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-15T00:47:19.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-15T00:47:19.034 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-15T00:47:19.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-15T00:47:19.036 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2024-09-15T00:47:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2639722496 bytes 2024-09-15T00:47:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2640334848 bytes 2024-09-15T00:47:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2646362112 bytes 2024-09-15T00:47:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2684771328 bytes 2024-09-15T00:47:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2685383680 bytes 2024-09-15T00:47:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2722623488 bytes 2024-09-15T00:47:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2762114048 bytes 2024-09-15T00:47:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2762738688 bytes 2024-09-15T00:47:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2808593408 bytes 2024-09-15T00:47:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2825870336 bytes 2024-09-15T00:47:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2854161408 bytes 2024-09-15T00:47:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2893020160 bytes 2024-09-15T00:47:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2933809152 bytes 2024-09-15T00:47:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2974449664 bytes 2024-09-15T00:47:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2990488576 bytes 2024-09-15T00:47:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3023223808 bytes 2024-09-15T00:47:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3095185408 bytes 2024-09-15T00:47:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3108629504 bytes 2024-09-15T00:47:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3178000384 bytes 2024-09-15T00:47:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3215152128 bytes 2024-09-15T00:47:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3250878464 bytes 2024-09-15T00:47:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3337067520 bytes 2024-09-15T00:47:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3355895808 bytes 2024-09-15T00:47:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3413100544 bytes 2024-09-15T00:47:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3463107584 bytes 2024-09-15T00:47:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3494579200 bytes 2024-09-15T00:47:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3608242176 bytes 2024-09-15T00:47:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3696145408 bytes 2024-09-15T00:47:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3731619840 bytes 2024-09-15T00:47:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3815422976 bytes 2024-09-15T00:47:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3828294656 bytes 2024-09-15T00:47:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3878589440 bytes 2024-09-15T00:47:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3914199040 bytes 2024-09-15T00:47:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4001935360 bytes 2024-09-15T00:47:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4021295104 bytes 2024-09-15T00:47:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4034305024 bytes 2024-09-15T00:47:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4110502912 bytes 2024-09-15T00:47:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4136058880 bytes 2024-09-15T00:47:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4162243584 bytes 2024-09-15T00:47:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4197663744 bytes 2024-09-15T00:47:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4255604736 bytes 2024-09-15T00:47:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4331081728 bytes 2024-09-15T00:47:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4388905984 bytes 2024-09-15T00:47:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4446426112 bytes 2024-09-15T00:47:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 4478267392 bytes 2024-09-15T00:47:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4495514624 bytes 2024-09-15T00:47:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 4539805696 bytes 2024-09-15T00:47:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 4600654848 bytes 2024-09-15T00:47:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 4605601792 bytes 2024-09-15T00:47:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11100 IOs, 4681831424 bytes 2024-09-15T00:47:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11200 IOs, 4686845952 bytes 2024-09-15T00:47:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11300 IOs, 4724149248 bytes 2024-09-15T00:47:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11400 IOs, 4753126400 bytes 2024-09-15T00:47:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11500 IOs, 4799880192 bytes 2024-09-15T00:47:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11600 IOs, 4820973568 bytes 2024-09-15T00:47:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11700 IOs, 4830263296 bytes 2024-09-15T00:47:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11800 IOs, 4945892352 bytes 2024-09-15T00:47:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11900 IOs, 4989770752 bytes 2024-09-15T00:47:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:47:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:47:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:47:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_5 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-15T00:47:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:47:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:47:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:47:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-15T00:47:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:47:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:47:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-15T00:47:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:47:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_5': 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 12a39c7d08c0 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.12a39c7d08c0 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:47:25 2024 2024-09-15T00:47:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:47:25 2024 2024-09-15T00:47:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:47:25 2024 2024-09-15T00:47:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:47:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:47:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 8d62f872-96bf-4d6b-b93e-efad6f76511e 2024-09-15T00:47:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:47:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:47:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-15T00:47:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:47:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:47:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=13 2024-09-15T00:47:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:47:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:47:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 22175744 bytes 2024-09-15T00:47:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 51739648 bytes 2024-09-15T00:47:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 197812224 bytes 2024-09-15T00:47:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 207750144 bytes 2024-09-15T00:47:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 221326336 bytes 2024-09-15T00:47:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 280693760 bytes 2024-09-15T00:47:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 339300352 bytes 2024-09-15T00:47:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 391328768 bytes 2024-09-15T00:47:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 437079040 bytes 2024-09-15T00:47:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 488250368 bytes 2024-09-15T00:47:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 529212416 bytes 2024-09-15T00:47:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 538418176 bytes 2024-09-15T00:47:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 560736256 bytes 2024-09-15T00:47:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 579989504 bytes 2024-09-15T00:47:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 623550464 bytes 2024-09-15T00:47:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 686691328 bytes 2024-09-15T00:47:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 778234880 bytes 2024-09-15T00:47:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 808813568 bytes 2024-09-15T00:47:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 825812992 bytes 2024-09-15T00:47:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 834147328 bytes 2024-09-15T00:47:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 880693248 bytes 2024-09-15T00:47:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 918742016 bytes 2024-09-15T00:47:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 919362560 bytes 2024-09-15T00:47:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 962277376 bytes 2024-09-15T00:47:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 981114880 bytes 2024-09-15T00:47:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1024822272 bytes 2024-09-15T00:47:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1074569216 bytes 2024-09-15T00:47:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1132760064 bytes 2024-09-15T00:47:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1294737408 bytes 2024-09-15T00:47:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1311232000 bytes 2024-09-15T00:47:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1314042880 bytes 2024-09-15T00:47:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1367019520 bytes 2024-09-15T00:47:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1472545792 bytes 2024-09-15T00:47:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1539288064 bytes 2024-09-15T00:47:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1560847360 bytes 2024-09-15T00:47:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1590924288 bytes 2024-09-15T00:47:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1591535616 bytes 2024-09-15T00:47:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1669203968 bytes 2024-09-15T00:47:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1750144000 bytes 2024-09-15T00:47:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1812874240 bytes 2024-09-15T00:47:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1867016192 bytes 2024-09-15T00:47:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1878117376 bytes 2024-09-15T00:47:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1899483136 bytes 2024-09-15T00:47:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1909110784 bytes 2024-09-15T00:47:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1949888512 bytes 2024-09-15T00:47:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1983951872 bytes 2024-09-15T00:47:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2027528192 bytes 2024-09-15T00:47:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2031958016 bytes 2024-09-15T00:47:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2077799424 bytes 2024-09-15T00:47:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2112415744 bytes 2024-09-15T00:47:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2117357568 bytes 2024-09-15T00:47:32.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2160545792 bytes 2024-09-15T00:47:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2218338304 bytes 2024-09-15T00:47:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2229811200 bytes 2024-09-15T00:47:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2238913536 bytes 2024-09-15T00:47:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2294596608 bytes 2024-09-15T00:47:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2300699648 bytes 2024-09-15T00:47:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2302997504 bytes 2024-09-15T00:47:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2365238272 bytes 2024-09-15T00:47:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2410285056 bytes 2024-09-15T00:47:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2470555648 bytes 2024-09-15T00:47:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2535689216 bytes 2024-09-15T00:47:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2581246976 bytes 2024-09-15T00:47:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2607858688 bytes 2024-09-15T00:47:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2637166592 bytes 2024-09-15T00:47:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2661926912 bytes 2024-09-15T00:47:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2724014080 bytes 2024-09-15T00:47:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2834449408 bytes 2024-09-15T00:47:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2900676608 bytes 2024-09-15T00:47:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2939318272 bytes 2024-09-15T00:47:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2960415744 bytes 2024-09-15T00:47:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2987210752 bytes 2024-09-15T00:47:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3009545216 bytes 2024-09-15T00:47:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3082840064 bytes 2024-09-15T00:47:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3107309568 bytes 2024-09-15T00:47:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3165137920 bytes 2024-09-15T00:47:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3205227520 bytes 2024-09-15T00:47:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3306183680 bytes 2024-09-15T00:47:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3308357632 bytes 2024-09-15T00:47:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3337487360 bytes 2024-09-15T00:47:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3403251712 bytes 2024-09-15T00:47:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3459177472 bytes 2024-09-15T00:47:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3514399744 bytes 2024-09-15T00:47:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3552354304 bytes 2024-09-15T00:47:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3665780736 bytes 2024-09-15T00:47:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3688926208 bytes 2024-09-15T00:47:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3689550848 bytes 2024-09-15T00:47:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3742586880 bytes 2024-09-15T00:47:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3808483328 bytes 2024-09-15T00:47:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3885230080 bytes 2024-09-15T00:47:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3962232832 bytes 2024-09-15T00:47:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3965407232 bytes 2024-09-15T00:47:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 4003299328 bytes 2024-09-15T00:47:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4020814848 bytes 2024-09-15T00:47:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4080819200 bytes 2024-09-15T00:47:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4119583744 bytes 2024-09-15T00:47:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4144199680 bytes 2024-09-15T00:47:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4236777472 bytes 2024-09-15T00:47:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4262132736 bytes 2024-09-15T00:47:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4269989888 bytes 2024-09-15T00:47:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4324963328 bytes 2024-09-15T00:47:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4365762560 bytes 2024-09-15T00:47:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4415724544 bytes 2024-09-15T00:47:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4489779200 bytes 2024-09-15T00:47:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4534146048 bytes 2024-09-15T00:47:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 4606838784 bytes 2024-09-15T00:47:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4630528000 bytes 2024-09-15T00:47:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 4706294784 bytes 2024-09-15T00:47:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 4770698240 bytes 2024-09-15T00:47:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 4838297600 bytes 2024-09-15T00:47:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11100 IOs, 4882573312 bytes 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_6 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:47:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:47:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:47:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-15T00:47:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:47:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:47:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:47:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-15T00:47:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:47:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:47:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-15T00:47:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:47:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-15T00:47:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_6': 2024-09-15T00:47:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 12af9e8d48c4 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.12af9e8d48c4 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:47:39 2024 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:47:39 2024 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:47:39 2024 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:47:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 003fd3d0-584f-46ad-a08c-185239d8066b 2024-09-15T00:47:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:47:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:47:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-15T00:47:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:47:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:47:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=5 2024-09-15T00:47:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:47:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:47:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 46240768 bytes 2024-09-15T00:47:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 86798336 bytes 2024-09-15T00:47:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 149572608 bytes 2024-09-15T00:47:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 184678400 bytes 2024-09-15T00:47:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 185287680 bytes 2024-09-15T00:47:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 221374464 bytes 2024-09-15T00:47:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 280856576 bytes 2024-09-15T00:47:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 329886720 bytes 2024-09-15T00:47:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 444210176 bytes 2024-09-15T00:47:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 511315968 bytes 2024-09-15T00:47:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 567035904 bytes 2024-09-15T00:47:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 655209472 bytes 2024-09-15T00:47:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 749268992 bytes 2024-09-15T00:47:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 839371776 bytes 2024-09-15T00:47:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 873387008 bytes 2024-09-15T00:47:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 900401152 bytes 2024-09-15T00:47:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 955301888 bytes 2024-09-15T00:47:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 982238208 bytes 2024-09-15T00:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 1008193536 bytes 2024-09-15T00:47:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1051163648 bytes 2024-09-15T00:47:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1092321280 bytes 2024-09-15T00:47:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1143320576 bytes 2024-09-15T00:47:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1181080576 bytes 2024-09-15T00:47:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1251046400 bytes 2024-09-15T00:47:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1297260544 bytes 2024-09-15T00:47:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1322068992 bytes 2024-09-15T00:47:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1328478208 bytes 2024-09-15T00:47:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1352827904 bytes 2024-09-15T00:47:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1359449088 bytes 2024-09-15T00:47:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1425814528 bytes 2024-09-15T00:47:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1452819456 bytes 2024-09-15T00:47:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1462037504 bytes 2024-09-15T00:47:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1503118336 bytes 2024-09-15T00:47:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1589903360 bytes 2024-09-15T00:47:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1657163776 bytes 2024-09-15T00:47:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1749905408 bytes 2024-09-15T00:47:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1819918336 bytes 2024-09-15T00:47:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1866513408 bytes 2024-09-15T00:47:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:47:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:47:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_7 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:47:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:47:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-15T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-15T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-15T00:47:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:47:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-15T00:47:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_7': 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 12bb25ef7dd 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.12bb25ef7dd 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:47:45 2024 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:47:45 2024 2024-09-15T00:47:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:47:45 2024 2024-09-15T00:47:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:47:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:47:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 35b7b6ec-da56-4f63-91b0-35afde53e670 2024-09-15T00:47:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-15T00:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:47:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=14 2024-09-15T00:47:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:47:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:47:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 43299840 bytes 2024-09-15T00:47:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 80001024 bytes 2024-09-15T00:47:46.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 130988032 bytes 2024-09-15T00:47:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 170157056 bytes 2024-09-15T00:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 202137600 bytes 2024-09-15T00:47:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 249905152 bytes 2024-09-15T00:47:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 283717632 bytes 2024-09-15T00:47:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 312558592 bytes 2024-09-15T00:47:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 353137664 bytes 2024-09-15T00:47:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 379025408 bytes 2024-09-15T00:47:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 480885760 bytes 2024-09-15T00:47:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 564790272 bytes 2024-09-15T00:47:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 588913664 bytes 2024-09-15T00:47:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 595922944 bytes 2024-09-15T00:47:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 639867904 bytes 2024-09-15T00:47:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 703945728 bytes 2024-09-15T00:47:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 745251840 bytes 2024-09-15T00:47:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 801421312 bytes 2024-09-15T00:47:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 823320576 bytes 2024-09-15T00:47:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 857288704 bytes 2024-09-15T00:47:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 919821312 bytes 2024-09-15T00:47:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 964971520 bytes 2024-09-15T00:47:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 983268352 bytes 2024-09-15T00:47:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1017913344 bytes 2024-09-15T00:47:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1045103616 bytes 2024-09-15T00:47:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1087432704 bytes 2024-09-15T00:47:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1093654528 bytes 2024-09-15T00:47:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1144338432 bytes 2024-09-15T00:47:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1208289280 bytes 2024-09-15T00:47:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1226112000 bytes 2024-09-15T00:47:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1279953920 bytes 2024-09-15T00:47:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1288268800 bytes 2024-09-15T00:47:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1371649024 bytes 2024-09-15T00:47:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1376866304 bytes 2024-09-15T00:47:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1443080192 bytes 2024-09-15T00:47:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1495974912 bytes 2024-09-15T00:47:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1527005184 bytes 2024-09-15T00:47:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1566462976 bytes 2024-09-15T00:47:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1612493824 bytes 2024-09-15T00:47:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1638211584 bytes 2024-09-15T00:47:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1706893312 bytes 2024-09-15T00:47:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1730887680 bytes 2024-09-15T00:47:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1773106176 bytes 2024-09-15T00:47:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1869429760 bytes 2024-09-15T00:47:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1888671744 bytes 2024-09-15T00:47:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1957863424 bytes 2024-09-15T00:47:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2007145472 bytes 2024-09-15T00:47:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2074713088 bytes 2024-09-15T00:47:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2118175744 bytes 2024-09-15T00:47:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2193046528 bytes 2024-09-15T00:47:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2213394432 bytes 2024-09-15T00:47:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2320105472 bytes 2024-09-15T00:47:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2344825856 bytes 2024-09-15T00:47:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2345435136 bytes 2024-09-15T00:47:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2398238720 bytes 2024-09-15T00:47:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2416484352 bytes 2024-09-15T00:47:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2466283520 bytes 2024-09-15T00:47:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2471655424 bytes 2024-09-15T00:47:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2493974528 bytes 2024-09-15T00:47:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2518273024 bytes 2024-09-15T00:47:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2582395904 bytes 2024-09-15T00:47:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2635480064 bytes 2024-09-15T00:47:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2636080128 bytes 2024-09-15T00:47:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2750681088 bytes 2024-09-15T00:47:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2793302016 bytes 2024-09-15T00:47:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2839219200 bytes 2024-09-15T00:47:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2924068864 bytes 2024-09-15T00:47:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2971654144 bytes 2024-09-15T00:47:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2996771840 bytes 2024-09-15T00:47:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3040561152 bytes 2024-09-15T00:47:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3077463040 bytes 2024-09-15T00:47:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3123426304 bytes 2024-09-15T00:47:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3139511296 bytes 2024-09-15T00:47:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3159925760 bytes 2024-09-15T00:47:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3217600512 bytes 2024-09-15T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3264060416 bytes 2024-09-15T00:47:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3294630912 bytes 2024-09-15T00:47:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3321913344 bytes 2024-09-15T00:47:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3389790208 bytes 2024-09-15T00:47:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3440311296 bytes 2024-09-15T00:47:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3478339584 bytes 2024-09-15T00:47:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3524600832 bytes 2024-09-15T00:47:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3597445120 bytes 2024-09-15T00:47:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3695329280 bytes 2024-09-15T00:47:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3718450176 bytes 2024-09-15T00:47:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3741957120 bytes 2024-09-15T00:47:56.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3763044352 bytes 2024-09-15T00:47:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3842029568 bytes 2024-09-15T00:47:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3854930944 bytes 2024-09-15T00:47:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3905040384 bytes 2024-09-15T00:47:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3905677312 bytes 2024-09-15T00:47:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3970422784 bytes 2024-09-15T00:47:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3997940736 bytes 2024-09-15T00:47:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4054904832 bytes 2024-09-15T00:47:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4098206720 bytes 2024-09-15T00:47:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4142896128 bytes 2024-09-15T00:47:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4157166592 bytes 2024-09-15T00:47:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4212622336 bytes 2024-09-15T00:47:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4240123904 bytes 2024-09-15T00:47:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4345827328 bytes 2024-09-15T00:47:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4434183168 bytes 2024-09-15T00:47:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4461582336 bytes 2024-09-15T00:47:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4534913024 bytes 2024-09-15T00:47:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4559875072 bytes 2024-09-15T00:47:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4566795264 bytes 2024-09-15T00:47:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 4629721088 bytes 2024-09-15T00:47:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4647398400 bytes 2024-09-15T00:47:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 4684715008 bytes 2024-09-15T00:47:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 4725680128 bytes 2024-09-15T00:47:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 4779489280 bytes 2024-09-15T00:47:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11100 IOs, 4889323520 bytes 2024-09-15T00:47:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11200 IOs, 4908332032 bytes 2024-09-15T00:47:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11300 IOs, 4942155776 bytes 2024-09-15T00:47:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11400 IOs, 4983478272 bytes 2024-09-15T00:47:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11500 IOs, 4984084480 bytes 2024-09-15T00:47:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11600 IOs, 4993264640 bytes 2024-09-15T00:47:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11700 IOs, 5038704640 bytes 2024-09-15T00:48:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11800 IOs, 5061261312 bytes 2024-09-15T00:48:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11900 IOs, 5097635840 bytes 2024-09-15T00:48:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12000 IOs, 5098248192 bytes 2024-09-15T00:48:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:12100 IOs, 5189198848 bytes 2024-09-15T00:48:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:48:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:48:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:48:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_8 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:48:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-15T00:48:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:48:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:48:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-15T00:48:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:48:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-15T00:48:01.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:48:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-15T00:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_8': 2024-09-15T00:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 12c762b3ccf3 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.12c762b3ccf3 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:00 2024 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:00 2024 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:00 2024 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 5728982c-d76d-410e-bc93-a9b0b0701e25 2024-09-15T00:48:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:48:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:48:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-15T00:48:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:48:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:48:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=7 2024-09-15T00:48:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:48:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:48:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 634880 bytes 2024-09-15T00:48:01.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 80670720 bytes 2024-09-15T00:48:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 98516992 bytes 2024-09-15T00:48:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 134216704 bytes 2024-09-15T00:48:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 291347456 bytes 2024-09-15T00:48:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 310626304 bytes 2024-09-15T00:48:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 350984192 bytes 2024-09-15T00:48:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 364897280 bytes 2024-09-15T00:48:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 388044800 bytes 2024-09-15T00:48:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 416025600 bytes 2024-09-15T00:48:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 438246400 bytes 2024-09-15T00:48:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 438888448 bytes 2024-09-15T00:48:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 445366272 bytes 2024-09-15T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 476550144 bytes 2024-09-15T00:48:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 586984448 bytes 2024-09-15T00:48:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 631677952 bytes 2024-09-15T00:48:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 677685248 bytes 2024-09-15T00:48:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 688276480 bytes 2024-09-15T00:48:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 688890880 bytes 2024-09-15T00:48:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 725450752 bytes 2024-09-15T00:48:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 765175808 bytes 2024-09-15T00:48:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 765813760 bytes 2024-09-15T00:48:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 786462720 bytes 2024-09-15T00:48:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 795199488 bytes 2024-09-15T00:48:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 812366848 bytes 2024-09-15T00:48:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 834830336 bytes 2024-09-15T00:48:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 875222016 bytes 2024-09-15T00:48:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 953812992 bytes 2024-09-15T00:48:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 995901440 bytes 2024-09-15T00:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1008894976 bytes 2024-09-15T00:48:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1063330816 bytes 2024-09-15T00:48:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1088476160 bytes 2024-09-15T00:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1137275904 bytes 2024-09-15T00:48:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1200804864 bytes 2024-09-15T00:48:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1230452736 bytes 2024-09-15T00:48:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1245540352 bytes 2024-09-15T00:48:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1316190208 bytes 2024-09-15T00:48:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1405737984 bytes 2024-09-15T00:48:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1434279936 bytes 2024-09-15T00:48:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1463555072 bytes 2024-09-15T00:48:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1486961664 bytes 2024-09-15T00:48:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1514242048 bytes 2024-09-15T00:48:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1629519872 bytes 2024-09-15T00:48:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1673358336 bytes 2024-09-15T00:48:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1694391296 bytes 2024-09-15T00:48:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1757129728 bytes 2024-09-15T00:48:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1844733952 bytes 2024-09-15T00:48:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1882932224 bytes 2024-09-15T00:48:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1887862784 bytes 2024-09-15T00:48:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1923523584 bytes 2024-09-15T00:48:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1946799104 bytes 2024-09-15T00:48:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 1981868032 bytes 2024-09-15T00:48:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2043638784 bytes 2024-09-15T00:48:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2044218368 bytes 2024-09-15T00:48:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2103999488 bytes 2024-09-15T00:48:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2160290816 bytes 2024-09-15T00:48:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2163205120 bytes 2024-09-15T00:48:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2194894848 bytes 2024-09-15T00:48:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2235087872 bytes 2024-09-15T00:48:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2277042176 bytes 2024-09-15T00:48:08.362 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:48:08.361+0000 7fc49cacc640 -1 reset not still connected to 0x560c4d706410 2024-09-15T00:48:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2307412992 bytes 2024-09-15T00:48:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:48:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:48:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:48:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:48:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_9 2024-09-15T00:48:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-15T00:48:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:48:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:48:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-15T00:48:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:48:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:48:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:48:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:48:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-15T00:48:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:48:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:48:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-15T00:48:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:48:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-15T00:48:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:48:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-15T00:48:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_9': 2024-09-15T00:48:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:48:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:48:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:48:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 12d3e06efe19 2024-09-15T00:48:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.12d3e06efe19 2024-09-15T00:48:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:48:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:48:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:48:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:48:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:08 2024 2024-09-15T00:48:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:08 2024 2024-09-15T00:48:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:08 2024 2024-09-15T00:48:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:48:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:48:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 344e6c94-9331-4900-a47f-b1036d651c77 2024-09-15T00:48:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:48:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:48:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-15T00:48:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:48:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=8 2024-09-15T00:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:48:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:48:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 28747776 bytes 2024-09-15T00:48:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 52258816 bytes 2024-09-15T00:48:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 66958336 bytes 2024-09-15T00:48:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 107603968 bytes 2024-09-15T00:48:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 164158464 bytes 2024-09-15T00:48:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 191842304 bytes 2024-09-15T00:48:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 225362944 bytes 2024-09-15T00:48:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 261137408 bytes 2024-09-15T00:48:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 310642688 bytes 2024-09-15T00:48:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 350078976 bytes 2024-09-15T00:48:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 376612864 bytes 2024-09-15T00:48:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 398091264 bytes 2024-09-15T00:48:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 445125632 bytes 2024-09-15T00:48:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 453453824 bytes 2024-09-15T00:48:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 484234240 bytes 2024-09-15T00:48:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 547276800 bytes 2024-09-15T00:48:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 577115136 bytes 2024-09-15T00:48:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 618689536 bytes 2024-09-15T00:48:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 682031104 bytes 2024-09-15T00:48:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 748231680 bytes 2024-09-15T00:48:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 786976768 bytes 2024-09-15T00:48:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 839078912 bytes 2024-09-15T00:48:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 900288512 bytes 2024-09-15T00:48:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 953414656 bytes 2024-09-15T00:48:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1035325440 bytes 2024-09-15T00:48:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1071089664 bytes 2024-09-15T00:48:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1138234368 bytes 2024-09-15T00:48:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1158746112 bytes 2024-09-15T00:48:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1208338432 bytes 2024-09-15T00:48:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1257571328 bytes 2024-09-15T00:48:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1285351424 bytes 2024-09-15T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1333800960 bytes 2024-09-15T00:48:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1337291776 bytes 2024-09-15T00:48:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1372121088 bytes 2024-09-15T00:48:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1405869056 bytes 2024-09-15T00:48:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1451656192 bytes 2024-09-15T00:48:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1476612096 bytes 2024-09-15T00:48:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1521105920 bytes 2024-09-15T00:48:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1581140992 bytes 2024-09-15T00:48:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1604769792 bytes 2024-09-15T00:48:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1640327168 bytes 2024-09-15T00:48:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1707563008 bytes 2024-09-15T00:48:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1778493440 bytes 2024-09-15T00:48:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1779093504 bytes 2024-09-15T00:48:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1821604864 bytes 2024-09-15T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1877415936 bytes 2024-09-15T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1926454272 bytes 2024-09-15T00:48:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1943308288 bytes 2024-09-15T00:48:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1966411776 bytes 2024-09-15T00:48:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1973344256 bytes 2024-09-15T00:48:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1986581504 bytes 2024-09-15T00:48:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2040939520 bytes 2024-09-15T00:48:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2074537984 bytes 2024-09-15T00:48:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2089133056 bytes 2024-09-15T00:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2209842176 bytes 2024-09-15T00:48:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2274626560 bytes 2024-09-15T00:48:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2297591808 bytes 2024-09-15T00:48:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2386275328 bytes 2024-09-15T00:48:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2413282304 bytes 2024-09-15T00:48:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2413894656 bytes 2024-09-15T00:48:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2485884928 bytes 2024-09-15T00:48:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2555780096 bytes 2024-09-15T00:48:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2576222208 bytes 2024-09-15T00:48:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2603262976 bytes 2024-09-15T00:48:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2633675776 bytes 2024-09-15T00:48:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2666654720 bytes 2024-09-15T00:48:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2699524096 bytes 2024-09-15T00:48:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:48:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:48:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:48:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:48:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_10 2024-09-15T00:48:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-15T00:48:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:48:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:48:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:48:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:48:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-15T00:48:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:48:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:48:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-15T00:48:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:48:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-15T00:48:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:48:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-15T00:48:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_10': 2024-09-15T00:48:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:48:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:48:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 12df171fd8a5 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.12df171fd8a5 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:17 2024 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:17 2024 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:17 2024 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 7902b9a5-d221-4297-9894-a93a67ef6ff7 2024-09-15T00:48:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:48:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:48:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-15T00:48:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:48:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:48:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=10 2024-09-15T00:48:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:48:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:48:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 4963328 bytes 2024-09-15T00:48:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 30230528 bytes 2024-09-15T00:48:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 113997824 bytes 2024-09-15T00:48:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 126657536 bytes 2024-09-15T00:48:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 160503808 bytes 2024-09-15T00:48:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 222200832 bytes 2024-09-15T00:48:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 278259712 bytes 2024-09-15T00:48:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 328334336 bytes 2024-09-15T00:48:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 413935616 bytes 2024-09-15T00:48:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 463623168 bytes 2024-09-15T00:48:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 527979520 bytes 2024-09-15T00:48:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 565560320 bytes 2024-09-15T00:48:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 625605632 bytes 2024-09-15T00:48:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 710070272 bytes 2024-09-15T00:48:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 734319616 bytes 2024-09-15T00:48:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 821837824 bytes 2024-09-15T00:48:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 858113024 bytes 2024-09-15T00:48:20.823 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-15T00:48:20.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-15T00:48:20.824 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-15T00:48:20.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T00:48:20.823+0000 7f68da729640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 41010) UID: 1000 2024-09-15T00:48:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 917202944 bytes 2024-09-15T00:48:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 972490752 bytes 2024-09-15T00:48:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1015846912 bytes 2024-09-15T00:48:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1035185152 bytes 2024-09-15T00:48:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1106552832 bytes 2024-09-15T00:48:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1140917248 bytes 2024-09-15T00:48:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1150357504 bytes 2024-09-15T00:48:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1183647744 bytes 2024-09-15T00:48:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1206176768 bytes 2024-09-15T00:48:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1287419904 bytes 2024-09-15T00:48:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1313063936 bytes 2024-09-15T00:48:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1355011072 bytes 2024-09-15T00:48:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1374349312 bytes 2024-09-15T00:48:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1431164928 bytes 2024-09-15T00:48:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1488463872 bytes 2024-09-15T00:48:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1552009216 bytes 2024-09-15T00:48:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1603896320 bytes 2024-09-15T00:48:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1682043904 bytes 2024-09-15T00:48:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1750785024 bytes 2024-09-15T00:48:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1782872064 bytes 2024-09-15T00:48:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1812316160 bytes 2024-09-15T00:48:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1834068992 bytes 2024-09-15T00:48:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1855503360 bytes 2024-09-15T00:48:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1868511232 bytes 2024-09-15T00:48:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1882997760 bytes 2024-09-15T00:48:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1916910592 bytes 2024-09-15T00:48:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2085901312 bytes 2024-09-15T00:48:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2129317888 bytes 2024-09-15T00:48:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2151777280 bytes 2024-09-15T00:48:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2201975808 bytes 2024-09-15T00:48:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2248509440 bytes 2024-09-15T00:48:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2266621952 bytes 2024-09-15T00:48:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2280031232 bytes 2024-09-15T00:48:24.825 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-15T00:48:24.825 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:48:24.825 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-15T00:48:24.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-15T00:48:24.826 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-15T00:48:24.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-15T00:48:24.829 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-15T00:48:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2313283584 bytes 2024-09-15T00:48:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2381270016 bytes 2024-09-15T00:48:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2447024128 bytes 2024-09-15T00:48:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2510039040 bytes 2024-09-15T00:48:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2515442688 bytes 2024-09-15T00:48:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2576046080 bytes 2024-09-15T00:48:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2626283520 bytes 2024-09-15T00:48:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2674876416 bytes 2024-09-15T00:48:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2675509248 bytes 2024-09-15T00:48:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2752102400 bytes 2024-09-15T00:48:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2789268480 bytes 2024-09-15T00:48:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2885091328 bytes 2024-09-15T00:48:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2936808448 bytes 2024-09-15T00:48:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2938789888 bytes 2024-09-15T00:48:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2983833600 bytes 2024-09-15T00:48:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 3034827776 bytes 2024-09-15T00:48:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3070670848 bytes 2024-09-15T00:48:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3079138304 bytes 2024-09-15T00:48:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3193870336 bytes 2024-09-15T00:48:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3259940864 bytes 2024-09-15T00:48:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3319080960 bytes 2024-09-15T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3332791296 bytes 2024-09-15T00:48:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3361092608 bytes 2024-09-15T00:48:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3366720512 bytes 2024-09-15T00:48:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3490363392 bytes 2024-09-15T00:48:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3573087232 bytes 2024-09-15T00:48:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3580220416 bytes 2024-09-15T00:48:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3633471488 bytes 2024-09-15T00:48:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:48:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:48:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:48:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:48:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_11 2024-09-15T00:48:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-15T00:48:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:48:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2024-09-15T00:48:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:48:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:48:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-15T00:48:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:48:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-15T00:48:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:48:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-15T00:48:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_11': 2024-09-15T00:48:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:48:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:48:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:48:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 12f0142723a5 2024-09-15T00:48:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.12f0142723a5 2024-09-15T00:48:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:48:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:48:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:48:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:48:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:28 2024 2024-09-15T00:48:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:28 2024 2024-09-15T00:48:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:28 2024 2024-09-15T00:48:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:48:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:48:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 4d7f7b50-55a3-40e9-8e7e-87fb1c8cf94e 2024-09-15T00:48:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:48:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:48:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-15T00:48:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:48:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:48:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=6 2024-09-15T00:48:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:48:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:48:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 619520 bytes 2024-09-15T00:48:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 29546496 bytes 2024-09-15T00:48:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 88709120 bytes 2024-09-15T00:48:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 145057792 bytes 2024-09-15T00:48:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 198855680 bytes 2024-09-15T00:48:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 243374080 bytes 2024-09-15T00:48:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 264127488 bytes 2024-09-15T00:48:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 295779328 bytes 2024-09-15T00:48:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 344271872 bytes 2024-09-15T00:48:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 392307712 bytes 2024-09-15T00:48:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 459763712 bytes 2024-09-15T00:48:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 519066624 bytes 2024-09-15T00:48:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 569085952 bytes 2024-09-15T00:48:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 597940224 bytes 2024-09-15T00:48:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 634810368 bytes 2024-09-15T00:48:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 668960768 bytes 2024-09-15T00:48:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 684508160 bytes 2024-09-15T00:48:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 708149248 bytes 2024-09-15T00:48:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 777798656 bytes 2024-09-15T00:48:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 881896448 bytes 2024-09-15T00:48:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 934398976 bytes 2024-09-15T00:48:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 951045120 bytes 2024-09-15T00:48:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1000199168 bytes 2024-09-15T00:48:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1054992384 bytes 2024-09-15T00:48:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1083772928 bytes 2024-09-15T00:48:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1130973184 bytes 2024-09-15T00:48:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1182258176 bytes 2024-09-15T00:48:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1226068992 bytes 2024-09-15T00:48:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1281056768 bytes 2024-09-15T00:48:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1322925056 bytes 2024-09-15T00:48:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1390379008 bytes 2024-09-15T00:48:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1441249280 bytes 2024-09-15T00:48:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1494066176 bytes 2024-09-15T00:48:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1577316352 bytes 2024-09-15T00:48:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1633654784 bytes 2024-09-15T00:48:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1684620288 bytes 2024-09-15T00:48:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1728531456 bytes 2024-09-15T00:48:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1766959104 bytes 2024-09-15T00:48:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1805249536 bytes 2024-09-15T00:48:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1911953408 bytes 2024-09-15T00:48:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1934196736 bytes 2024-09-15T00:48:34.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 2008580096 bytes 2024-09-15T00:48:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2038005760 bytes 2024-09-15T00:48:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2098758656 bytes 2024-09-15T00:48:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2120377344 bytes 2024-09-15T00:48:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2133739520 bytes 2024-09-15T00:48:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2188122112 bytes 2024-09-15T00:48:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2242439168 bytes 2024-09-15T00:48:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:48:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:48:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:48:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:48:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_12 2024-09-15T00:48:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-15T00:48:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:48:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:48:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-15T00:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:48:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:48:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2024-09-15T00:48:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:48:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:48:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-15T00:48:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:48:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-15T00:48:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:48:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-15T00:48:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_12': 2024-09-15T00:48:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:48:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:48:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 12fca06af045 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.12fca06af045 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:35 2024 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:35 2024 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:35 2024 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: a1914ce5-652f-465e-9c3f-e18ac5ac2ca8 2024-09-15T00:48:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:48:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:48:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-15T00:48:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:48:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:48:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=7 2024-09-15T00:48:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:48:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:48:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 50832384 bytes 2024-09-15T00:48:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 81178624 bytes 2024-09-15T00:48:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 116413440 bytes 2024-09-15T00:48:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 190731264 bytes 2024-09-15T00:48:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 227815424 bytes 2024-09-15T00:48:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 253096960 bytes 2024-09-15T00:48:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 332295168 bytes 2024-09-15T00:48:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 367383552 bytes 2024-09-15T00:48:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 367990784 bytes 2024-09-15T00:48:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 406453248 bytes 2024-09-15T00:48:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 444438528 bytes 2024-09-15T00:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 494914560 bytes 2024-09-15T00:48:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 550387712 bytes 2024-09-15T00:48:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 573906944 bytes 2024-09-15T00:48:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 642875392 bytes 2024-09-15T00:48:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 691445760 bytes 2024-09-15T00:48:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 700490752 bytes 2024-09-15T00:48:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 701113344 bytes 2024-09-15T00:48:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 763073536 bytes 2024-09-15T00:48:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 842007552 bytes 2024-09-15T00:48:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 864508928 bytes 2024-09-15T00:48:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 926347264 bytes 2024-09-15T00:48:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 953786368 bytes 2024-09-15T00:48:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 974856192 bytes 2024-09-15T00:48:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1013035008 bytes 2024-09-15T00:48:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1068359680 bytes 2024-09-15T00:48:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1117434880 bytes 2024-09-15T00:48:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1228532736 bytes 2024-09-15T00:48:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1277419520 bytes 2024-09-15T00:48:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1319809024 bytes 2024-09-15T00:48:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1393917952 bytes 2024-09-15T00:48:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1433202688 bytes 2024-09-15T00:48:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1504294912 bytes 2024-09-15T00:48:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1541782528 bytes 2024-09-15T00:48:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1573226496 bytes 2024-09-15T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1573855232 bytes 2024-09-15T00:48:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1634195456 bytes 2024-09-15T00:48:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1676749824 bytes 2024-09-15T00:48:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1801552896 bytes 2024-09-15T00:48:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1886534656 bytes 2024-09-15T00:48:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1946465280 bytes 2024-09-15T00:48:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1963074560 bytes 2024-09-15T00:48:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1988139008 bytes 2024-09-15T00:48:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2004349952 bytes 2024-09-15T00:48:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2041152512 bytes 2024-09-15T00:48:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2129442816 bytes 2024-09-15T00:48:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2151914496 bytes 2024-09-15T00:48:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2193421312 bytes 2024-09-15T00:48:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2208177152 bytes 2024-09-15T00:48:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2255156224 bytes 2024-09-15T00:48:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2283998208 bytes 2024-09-15T00:48:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2363792384 bytes 2024-09-15T00:48:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2418511872 bytes 2024-09-15T00:48:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2443343872 bytes 2024-09-15T00:48:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2503453696 bytes 2024-09-15T00:48:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2545576960 bytes 2024-09-15T00:48:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2561654784 bytes 2024-09-15T00:48:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:48:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:48:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:48:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:48:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_13 2024-09-15T00:48:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-15T00:48:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:48:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:48:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-15T00:48:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:48:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:48:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:48:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:48:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2024-09-15T00:48:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:48:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:48:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-15T00:48:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:48:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-15T00:48:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:48:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_13': 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13089ab41b02 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13089ab41b02 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:48:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:43 2024 2024-09-15T00:48:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:43 2024 2024-09-15T00:48:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:43 2024 2024-09-15T00:48:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:48:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:48:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 66207760-d8d4-44f5-80a1-fdab006bbc16 2024-09-15T00:48:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:48:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:48:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-15T00:48:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:48:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:48:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=12 2024-09-15T00:48:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:48:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:48:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 102753280 bytes 2024-09-15T00:48:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 116062208 bytes 2024-09-15T00:48:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 153352192 bytes 2024-09-15T00:48:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 190278656 bytes 2024-09-15T00:48:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 250295296 bytes 2024-09-15T00:48:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 314721280 bytes 2024-09-15T00:48:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 348266496 bytes 2024-09-15T00:48:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 373394432 bytes 2024-09-15T00:48:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 489274368 bytes 2024-09-15T00:48:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 530277376 bytes 2024-09-15T00:48:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 554397696 bytes 2024-09-15T00:48:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 554995712 bytes 2024-09-15T00:48:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 625043456 bytes 2024-09-15T00:48:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 638303232 bytes 2024-09-15T00:48:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 673827840 bytes 2024-09-15T00:48:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 695030784 bytes 2024-09-15T00:48:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 727131136 bytes 2024-09-15T00:48:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 820984832 bytes 2024-09-15T00:48:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 950573056 bytes 2024-09-15T00:48:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 968983552 bytes 2024-09-15T00:48:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1076292608 bytes 2024-09-15T00:48:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1092716544 bytes 2024-09-15T00:48:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1162712064 bytes 2024-09-15T00:48:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1177404416 bytes 2024-09-15T00:48:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1267051520 bytes 2024-09-15T00:48:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1300868096 bytes 2024-09-15T00:48:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1337406464 bytes 2024-09-15T00:48:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1397297152 bytes 2024-09-15T00:48:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1475968000 bytes 2024-09-15T00:48:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1495529472 bytes 2024-09-15T00:48:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1594829824 bytes 2024-09-15T00:48:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1680937984 bytes 2024-09-15T00:48:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1766330368 bytes 2024-09-15T00:48:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1781553152 bytes 2024-09-15T00:48:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1851103232 bytes 2024-09-15T00:48:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1893704704 bytes 2024-09-15T00:48:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1921866752 bytes 2024-09-15T00:48:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1948485632 bytes 2024-09-15T00:48:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1986430976 bytes 2024-09-15T00:48:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 2010936320 bytes 2024-09-15T00:48:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 2012700672 bytes 2024-09-15T00:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 2022936576 bytes 2024-09-15T00:48:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2072893440 bytes 2024-09-15T00:48:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2131203072 bytes 2024-09-15T00:48:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2167592960 bytes 2024-09-15T00:48:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2193037312 bytes 2024-09-15T00:48:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2199789568 bytes 2024-09-15T00:48:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2267452416 bytes 2024-09-15T00:48:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2333786112 bytes 2024-09-15T00:48:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2364312576 bytes 2024-09-15T00:48:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2399628288 bytes 2024-09-15T00:48:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2416716800 bytes 2024-09-15T00:48:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2432524288 bytes 2024-09-15T00:48:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2462968832 bytes 2024-09-15T00:48:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2463579136 bytes 2024-09-15T00:48:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2544972800 bytes 2024-09-15T00:48:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2554847232 bytes 2024-09-15T00:48:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2598029312 bytes 2024-09-15T00:48:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2617336832 bytes 2024-09-15T00:48:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2644017152 bytes 2024-09-15T00:48:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2717774848 bytes 2024-09-15T00:48:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2750823424 bytes 2024-09-15T00:48:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2800351232 bytes 2024-09-15T00:48:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2895121408 bytes 2024-09-15T00:48:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2918596608 bytes 2024-09-15T00:48:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2924250112 bytes 2024-09-15T00:48:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2943973376 bytes 2024-09-15T00:48:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3015159808 bytes 2024-09-15T00:48:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3028331520 bytes 2024-09-15T00:48:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3102137344 bytes 2024-09-15T00:48:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3143761920 bytes 2024-09-15T00:48:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3168484352 bytes 2024-09-15T00:48:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3229111296 bytes 2024-09-15T00:48:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3286342656 bytes 2024-09-15T00:48:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3296273408 bytes 2024-09-15T00:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3337335808 bytes 2024-09-15T00:48:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3337962496 bytes 2024-09-15T00:48:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3359322112 bytes 2024-09-15T00:48:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3380568064 bytes 2024-09-15T00:48:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3432449024 bytes 2024-09-15T00:48:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3448600576 bytes 2024-09-15T00:48:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3454654464 bytes 2024-09-15T00:48:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3484402688 bytes 2024-09-15T00:48:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3566196736 bytes 2024-09-15T00:48:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3614510080 bytes 2024-09-15T00:48:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3673588736 bytes 2024-09-15T00:48:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3761113088 bytes 2024-09-15T00:48:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3821256704 bytes 2024-09-15T00:48:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3844961280 bytes 2024-09-15T00:48:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3899364352 bytes 2024-09-15T00:48:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3933587456 bytes 2024-09-15T00:48:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3965996032 bytes 2024-09-15T00:48:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3977703424 bytes 2024-09-15T00:48:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4027683840 bytes 2024-09-15T00:48:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4065457152 bytes 2024-09-15T00:48:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4146433024 bytes 2024-09-15T00:48:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4183316480 bytes 2024-09-15T00:48:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4243082240 bytes 2024-09-15T00:48:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4264606720 bytes 2024-09-15T00:48:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:48:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:48:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:48:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_14 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-15T00:48:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:48:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:48:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:48:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:48:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:48:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2024-09-15T00:48:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:48:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:48:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:48:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-15T00:48:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:48:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:48:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-15T00:48:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:48:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-15T00:48:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_14': 2024-09-15T00:48:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:48:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:48:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:48:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1314bf320eaf 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1314bf320eaf 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:56 2024 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:56 2024 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:56 2024 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 5a261e22-728a-47f5-ad1c-cec823ae7b9a 2024-09-15T00:48:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:48:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:48:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-15T00:48:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:48:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:48:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:48:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=6 2024-09-15T00:48:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:48:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:48:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 602112 bytes 2024-09-15T00:48:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 83414016 bytes 2024-09-15T00:48:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 90588160 bytes 2024-09-15T00:48:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 147026944 bytes 2024-09-15T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 207642624 bytes 2024-09-15T00:48:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 289865728 bytes 2024-09-15T00:48:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 326072320 bytes 2024-09-15T00:48:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 346094592 bytes 2024-09-15T00:48:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 428480512 bytes 2024-09-15T00:48:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 474693632 bytes 2024-09-15T00:48:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 501761024 bytes 2024-09-15T00:48:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 544964608 bytes 2024-09-15T00:48:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 581305344 bytes 2024-09-15T00:48:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 641802240 bytes 2024-09-15T00:48:59.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 655921152 bytes 2024-09-15T00:48:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 656540672 bytes 2024-09-15T00:48:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 716015616 bytes 2024-09-15T00:48:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 728145920 bytes 2024-09-15T00:48:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 763789312 bytes 2024-09-15T00:48:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 798061568 bytes 2024-09-15T00:49:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 798705664 bytes 2024-09-15T00:49:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 859280384 bytes 2024-09-15T00:49:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 875445248 bytes 2024-09-15T00:49:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 885724160 bytes 2024-09-15T00:49:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 951770112 bytes 2024-09-15T00:49:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 969984000 bytes 2024-09-15T00:49:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1005728768 bytes 2024-09-15T00:49:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1047219200 bytes 2024-09-15T00:49:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1080041472 bytes 2024-09-15T00:49:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1128353792 bytes 2024-09-15T00:49:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1234882560 bytes 2024-09-15T00:49:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1297410048 bytes 2024-09-15T00:49:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1298045952 bytes 2024-09-15T00:49:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1352237056 bytes 2024-09-15T00:49:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1396788224 bytes 2024-09-15T00:49:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1413242880 bytes 2024-09-15T00:49:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1451560960 bytes 2024-09-15T00:49:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1486810112 bytes 2024-09-15T00:49:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1499021312 bytes 2024-09-15T00:49:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1513021440 bytes 2024-09-15T00:49:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1540558848 bytes 2024-09-15T00:49:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1611318272 bytes 2024-09-15T00:49:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1645478912 bytes 2024-09-15T00:49:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1669248000 bytes 2024-09-15T00:49:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1703023616 bytes 2024-09-15T00:49:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1730359296 bytes 2024-09-15T00:49:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1805821952 bytes 2024-09-15T00:49:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1818081280 bytes 2024-09-15T00:49:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1848956928 bytes 2024-09-15T00:49:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1937832960 bytes 2024-09-15T00:49:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:49:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:49:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:49:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:49:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:49:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_15 2024-09-15T00:49:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-15T00:49:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:49:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:49:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:49:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-15T00:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2024-09-15T00:49:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:49:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:49:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:49:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-15T00:49:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:49:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:49:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-15T00:49:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:49:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-15T00:49:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_15': 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 132089dc6f4b 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.132089dc6f4b 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:49:03 2024 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:49:03 2024 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:49:03 2024 2024-09-15T00:49:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:49:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:49:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 0df0e3a0-f71b-4bbc-8631-00ed1ea29cb7 2024-09-15T00:49:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:49:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:49:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-15T00:49:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:49:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:49:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=9 2024-09-15T00:49:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:49:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:49:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 85566464 bytes 2024-09-15T00:49:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 146906112 bytes 2024-09-15T00:49:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 155763712 bytes 2024-09-15T00:49:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 208712704 bytes 2024-09-15T00:49:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 228076544 bytes 2024-09-15T00:49:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 269254656 bytes 2024-09-15T00:49:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 288672768 bytes 2024-09-15T00:49:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 332721152 bytes 2024-09-15T00:49:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 362731520 bytes 2024-09-15T00:49:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 367285248 bytes 2024-09-15T00:49:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 415950848 bytes 2024-09-15T00:49:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 472925184 bytes 2024-09-15T00:49:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 519651328 bytes 2024-09-15T00:49:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 549774336 bytes 2024-09-15T00:49:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 569976832 bytes 2024-09-15T00:49:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 592742400 bytes 2024-09-15T00:49:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 610748416 bytes 2024-09-15T00:49:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 699548672 bytes 2024-09-15T00:49:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 719209472 bytes 2024-09-15T00:49:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 782338048 bytes 2024-09-15T00:49:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 823293952 bytes 2024-09-15T00:49:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 899286016 bytes 2024-09-15T00:49:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 984394752 bytes 2024-09-15T00:49:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1049069568 bytes 2024-09-15T00:49:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1084940288 bytes 2024-09-15T00:49:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1096876032 bytes 2024-09-15T00:49:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1122215936 bytes 2024-09-15T00:49:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1145230336 bytes 2024-09-15T00:49:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1200278528 bytes 2024-09-15T00:49:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1228223488 bytes 2024-09-15T00:49:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1300770816 bytes 2024-09-15T00:49:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1352016896 bytes 2024-09-15T00:49:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1352674304 bytes 2024-09-15T00:49:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1398777856 bytes 2024-09-15T00:49:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1475286016 bytes 2024-09-15T00:49:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1497883648 bytes 2024-09-15T00:49:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1546271744 bytes 2024-09-15T00:49:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1553270784 bytes 2024-09-15T00:49:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1584060416 bytes 2024-09-15T00:49:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1603403776 bytes 2024-09-15T00:49:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1633837056 bytes 2024-09-15T00:49:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1669056512 bytes 2024-09-15T00:49:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1754162176 bytes 2024-09-15T00:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1837306880 bytes 2024-09-15T00:49:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1893895168 bytes 2024-09-15T00:49:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1909241856 bytes 2024-09-15T00:49:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1946993664 bytes 2024-09-15T00:49:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1986969600 bytes 2024-09-15T00:49:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2038588416 bytes 2024-09-15T00:49:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2074686464 bytes 2024-09-15T00:49:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2092248064 bytes 2024-09-15T00:49:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2179304448 bytes 2024-09-15T00:49:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2197360640 bytes 2024-09-15T00:49:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2197979136 bytes 2024-09-15T00:49:11.037 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2532882798017624) > weight (0.25) 2024-09-15T00:49:11.037 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.31528989765422766) > weight (0.25) 2024-09-15T00:49:11.037 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-15T00:49:11.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-15T00:49:11.038 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-15T00:49:11.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T00:49:11.037+0000 7f62ebe87640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 45377) UID: 1000 2024-09-15T00:49:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2232626176 bytes 2024-09-15T00:49:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2282710016 bytes 2024-09-15T00:49:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2313377792 bytes 2024-09-15T00:49:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2352017408 bytes 2024-09-15T00:49:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2400671744 bytes 2024-09-15T00:49:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2414413824 bytes 2024-09-15T00:49:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2439660544 bytes 2024-09-15T00:49:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2518526976 bytes 2024-09-15T00:49:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2611051520 bytes 2024-09-15T00:49:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2667197440 bytes 2024-09-15T00:49:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2701359104 bytes 2024-09-15T00:49:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2711465984 bytes 2024-09-15T00:49:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2746350592 bytes 2024-09-15T00:49:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2793123840 bytes 2024-09-15T00:49:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2793749504 bytes 2024-09-15T00:49:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2836081664 bytes 2024-09-15T00:49:13.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2867590144 bytes 2024-09-15T00:49:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2892808192 bytes 2024-09-15T00:49:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2905480192 bytes 2024-09-15T00:49:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2958567424 bytes 2024-09-15T00:49:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3014144000 bytes 2024-09-15T00:49:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3049690112 bytes 2024-09-15T00:49:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:49:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:49:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:49:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:49:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:49:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_16 2024-09-15T00:49:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-15T00:49:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:49:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:49:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:49:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-15T00:49:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:49:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:49:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:49:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:49:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2024-09-15T00:49:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:49:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:49:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:49:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-15T00:49:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:49:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:49:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-15T00:49:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:49:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-15T00:49:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_16': 2024-09-15T00:49:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:49:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:49:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 132c8ecc2e93 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.132c8ecc2e93 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:49:13 2024 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:49:13 2024 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:49:13 2024 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: b1a838da-8633-4b6c-a717-7c58db5d3618 2024-09-15T00:49:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:49:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:49:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-15T00:49:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:49:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:49:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=12 2024-09-15T00:49:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:49:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:49:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 35107840 bytes 2024-09-15T00:49:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 116713472 bytes 2024-09-15T00:49:15.039 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-15T00:49:15.039 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:49:15.039 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-15T00:49:15.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-15T00:49:15.040 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-15T00:49:15.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-15T00:49:15.043 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-09-15T00:49:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 186460160 bytes 2024-09-15T00:49:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 211002368 bytes 2024-09-15T00:49:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 325099520 bytes 2024-09-15T00:49:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 325719040 bytes 2024-09-15T00:49:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 349234176 bytes 2024-09-15T00:49:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 370790400 bytes 2024-09-15T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 420874240 bytes 2024-09-15T00:49:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 471980032 bytes 2024-09-15T00:49:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 494436352 bytes 2024-09-15T00:49:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 546571264 bytes 2024-09-15T00:49:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 601356288 bytes 2024-09-15T00:49:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 672702464 bytes 2024-09-15T00:49:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 718968832 bytes 2024-09-15T00:49:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 791928832 bytes 2024-09-15T00:49:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 855823360 bytes 2024-09-15T00:49:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 945261568 bytes 2024-09-15T00:49:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 998443008 bytes 2024-09-15T00:49:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1053856768 bytes 2024-09-15T00:49:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1107662848 bytes 2024-09-15T00:49:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1148989440 bytes 2024-09-15T00:49:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1169427456 bytes 2024-09-15T00:49:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1197073408 bytes 2024-09-15T00:49:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1218121728 bytes 2024-09-15T00:49:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1238857728 bytes 2024-09-15T00:49:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1241975808 bytes 2024-09-15T00:49:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1257967616 bytes 2024-09-15T00:49:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1288716288 bytes 2024-09-15T00:49:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1327659008 bytes 2024-09-15T00:49:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1397113856 bytes 2024-09-15T00:49:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1397731328 bytes 2024-09-15T00:49:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1454051328 bytes 2024-09-15T00:49:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1557796864 bytes 2024-09-15T00:49:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1581596672 bytes 2024-09-15T00:49:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1659669504 bytes 2024-09-15T00:49:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1689093120 bytes 2024-09-15T00:49:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1721447424 bytes 2024-09-15T00:49:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1746099200 bytes 2024-09-15T00:49:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1801200640 bytes 2024-09-15T00:49:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1840046080 bytes 2024-09-15T00:49:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1952399360 bytes 2024-09-15T00:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2024725504 bytes 2024-09-15T00:49:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2062477312 bytes 2024-09-15T00:49:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2108343296 bytes 2024-09-15T00:49:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2150889472 bytes 2024-09-15T00:49:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2213985280 bytes 2024-09-15T00:49:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2244195328 bytes 2024-09-15T00:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2256247808 bytes 2024-09-15T00:49:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2266266624 bytes 2024-09-15T00:49:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2359582720 bytes 2024-09-15T00:49:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2378244096 bytes 2024-09-15T00:49:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2417030144 bytes 2024-09-15T00:49:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2459526144 bytes 2024-09-15T00:49:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2533090304 bytes 2024-09-15T00:49:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2567876608 bytes 2024-09-15T00:49:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2598159360 bytes 2024-09-15T00:49:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2622848000 bytes 2024-09-15T00:49:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2693828608 bytes 2024-09-15T00:49:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2743748608 bytes 2024-09-15T00:49:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2793171968 bytes 2024-09-15T00:49:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2794224640 bytes 2024-09-15T00:49:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2876036096 bytes 2024-09-15T00:49:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2903378944 bytes 2024-09-15T00:49:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2920483840 bytes 2024-09-15T00:49:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 3002968064 bytes 2024-09-15T00:49:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3052322816 bytes 2024-09-15T00:49:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3091923968 bytes 2024-09-15T00:49:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3171958784 bytes 2024-09-15T00:49:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3217682432 bytes 2024-09-15T00:49:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3259463680 bytes 2024-09-15T00:49:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3298513920 bytes 2024-09-15T00:49:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3365055488 bytes 2024-09-15T00:49:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3392254976 bytes 2024-09-15T00:49:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3450858496 bytes 2024-09-15T00:49:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3477529600 bytes 2024-09-15T00:49:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3537514496 bytes 2024-09-15T00:49:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3551119360 bytes 2024-09-15T00:49:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3608526848 bytes 2024-09-15T00:49:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3645680640 bytes 2024-09-15T00:49:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3711461376 bytes 2024-09-15T00:49:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3724605440 bytes 2024-09-15T00:49:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3762079744 bytes 2024-09-15T00:49:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3785729024 bytes 2024-09-15T00:49:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3891303424 bytes 2024-09-15T00:49:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3942625280 bytes 2024-09-15T00:49:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3972563968 bytes 2024-09-15T00:49:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 4042991616 bytes 2024-09-15T00:49:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 4100376576 bytes 2024-09-15T00:49:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 4213713920 bytes 2024-09-15T00:49:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 4259065856 bytes 2024-09-15T00:49:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 4356721664 bytes 2024-09-15T00:49:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 4452643840 bytes 2024-09-15T00:49:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:49:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:49:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:49:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:49:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:49:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_17 2024-09-15T00:49:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-15T00:49:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:49:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:49:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:49:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-15T00:49:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:49:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:49:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:49:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:49:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2024-09-15T00:49:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:49:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:49:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:49:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-15T00:49:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:49:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:49:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-15T00:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:49:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-15T00:49:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_17': 2024-09-15T00:49:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:49:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:49:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:49:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1341e36d1154 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1341e36d1154 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:49:26 2024 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:49:26 2024 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:49:26 2024 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 9a3b2ff6-9972-4b36-a5a7-25c8321572a0 2024-09-15T00:49:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:49:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:49:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-15T00:49:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:49:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:49:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=10 2024-09-15T00:49:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:49:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:49:27.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 13775872 bytes 2024-09-15T00:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 60312576 bytes 2024-09-15T00:49:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 106720256 bytes 2024-09-15T00:49:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 140484608 bytes 2024-09-15T00:49:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 195704832 bytes 2024-09-15T00:49:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 271667200 bytes 2024-09-15T00:49:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 333753344 bytes 2024-09-15T00:49:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 380787712 bytes 2024-09-15T00:49:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 433673216 bytes 2024-09-15T00:49:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 454542336 bytes 2024-09-15T00:49:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 512948224 bytes 2024-09-15T00:49:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 549225472 bytes 2024-09-15T00:49:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 580958208 bytes 2024-09-15T00:49:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 620567552 bytes 2024-09-15T00:49:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 677409792 bytes 2024-09-15T00:49:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 698339328 bytes 2024-09-15T00:49:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 719920128 bytes 2024-09-15T00:49:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 732206080 bytes 2024-09-15T00:49:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 760342528 bytes 2024-09-15T00:49:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 785652736 bytes 2024-09-15T00:49:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 809432064 bytes 2024-09-15T00:49:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 825931776 bytes 2024-09-15T00:49:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 861470720 bytes 2024-09-15T00:49:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 892368896 bytes 2024-09-15T00:49:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 923119616 bytes 2024-09-15T00:49:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 965398528 bytes 2024-09-15T00:49:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 989029376 bytes 2024-09-15T00:49:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1018770432 bytes 2024-09-15T00:49:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1026598912 bytes 2024-09-15T00:49:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1060016128 bytes 2024-09-15T00:49:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1113319424 bytes 2024-09-15T00:49:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1176962048 bytes 2024-09-15T00:49:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1227914240 bytes 2024-09-15T00:49:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1307943936 bytes 2024-09-15T00:49:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1360578560 bytes 2024-09-15T00:49:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1433469952 bytes 2024-09-15T00:49:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1491933184 bytes 2024-09-15T00:49:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1551868928 bytes 2024-09-15T00:49:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1615255552 bytes 2024-09-15T00:49:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1649416192 bytes 2024-09-15T00:49:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1687460864 bytes 2024-09-15T00:49:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1711173632 bytes 2024-09-15T00:49:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1731927040 bytes 2024-09-15T00:49:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1839819776 bytes 2024-09-15T00:49:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1894751232 bytes 2024-09-15T00:49:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2020107264 bytes 2024-09-15T00:49:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2058906624 bytes 2024-09-15T00:49:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2152811520 bytes 2024-09-15T00:49:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2226798592 bytes 2024-09-15T00:49:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2283086848 bytes 2024-09-15T00:49:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2326409216 bytes 2024-09-15T00:49:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2376972288 bytes 2024-09-15T00:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2428043264 bytes 2024-09-15T00:49:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2471148544 bytes 2024-09-15T00:49:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2500868096 bytes 2024-09-15T00:49:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2518688768 bytes 2024-09-15T00:49:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2533385216 bytes 2024-09-15T00:49:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2555908096 bytes 2024-09-15T00:49:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2647735296 bytes 2024-09-15T00:49:35.066 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:49:35.063+0000 7fc49cacc640 -1 reset not still connected to 0x560c58b9c1a0 2024-09-15T00:49:35.067 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:49:35.065+0000 7fc49cacc640 -1 reset not still connected to 0x560c59eaed00 2024-09-15T00:49:35.067 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:49:35.065+0000 7fc49cacc640 -1 reset not still connected to 0x560c5c381860 2024-09-15T00:49:35.071 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:49:35.069+0000 7fc49cacc640 -1 reset not still connected to 0x560c5d7e7110 2024-09-15T00:49:35.071 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:49:35.069+0000 7fc49cacc640 -1 reset not still connected to 0x560c62eb11e0 2024-09-15T00:49:35.071 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:49:35.069+0000 7fc49cacc640 -1 reset not still connected to 0x560c62ec95f0 2024-09-15T00:49:35.072 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:49:35.070+0000 7fc49cacc640 -1 reset not still connected to 0x560c6bbd3380 2024-09-15T00:49:35.072 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:49:35.070+0000 7fc49cacc640 -1 reset not still connected to 0x560c6c7ced00 2024-09-15T00:49:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2656992256 bytes 2024-09-15T00:49:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2678433792 bytes 2024-09-15T00:49:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2775919616 bytes 2024-09-15T00:49:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2808410112 bytes 2024-09-15T00:49:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2858864640 bytes 2024-09-15T00:49:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2887475200 bytes 2024-09-15T00:49:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2974608384 bytes 2024-09-15T00:49:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3030597632 bytes 2024-09-15T00:49:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3061294080 bytes 2024-09-15T00:49:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3100445696 bytes 2024-09-15T00:49:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3105068032 bytes 2024-09-15T00:49:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3138589696 bytes 2024-09-15T00:49:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3198810112 bytes 2024-09-15T00:49:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3264430080 bytes 2024-09-15T00:49:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3298723840 bytes 2024-09-15T00:49:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3328674816 bytes 2024-09-15T00:49:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3338583040 bytes 2024-09-15T00:49:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3358744576 bytes 2024-09-15T00:49:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3404274688 bytes 2024-09-15T00:49:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3436713984 bytes 2024-09-15T00:49:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3487074304 bytes 2024-09-15T00:49:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:49:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:49:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:49:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:49:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:49:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_18 2024-09-15T00:49:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-15T00:49:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:49:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:49:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:49:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:49:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2024-09-15T00:49:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:49:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:49:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:49:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-15T00:49:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:49:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:49:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-15T00:49:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:49:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-15T00:49:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_18': 2024-09-15T00:49:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:49:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:49:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:49:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 134dfb08fac9 2024-09-15T00:49:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.134dfb08fac9 2024-09-15T00:49:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:49:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:49:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:49:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:49:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:49:37 2024 2024-09-15T00:49:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:49:37 2024 2024-09-15T00:49:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:49:37 2024 2024-09-15T00:49:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:49:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:49:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 8c63bcd3-d5d8-46ca-9685-256e72243858 2024-09-15T00:49:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:49:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:49:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-15T00:49:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:49:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:49:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=11 2024-09-15T00:49:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:49:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:49:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 47453184 bytes 2024-09-15T00:49:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 135577600 bytes 2024-09-15T00:49:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 160939008 bytes 2024-09-15T00:49:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 211807232 bytes 2024-09-15T00:49:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 242411520 bytes 2024-09-15T00:49:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 248773632 bytes 2024-09-15T00:49:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 322867200 bytes 2024-09-15T00:49:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 441794560 bytes 2024-09-15T00:49:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 482828288 bytes 2024-09-15T00:49:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 514677760 bytes 2024-09-15T00:49:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 557419520 bytes 2024-09-15T00:49:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 558020608 bytes 2024-09-15T00:49:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 602130432 bytes 2024-09-15T00:49:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 668645376 bytes 2024-09-15T00:49:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 703354880 bytes 2024-09-15T00:49:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 803500032 bytes 2024-09-15T00:49:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 843787264 bytes 2024-09-15T00:49:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 942955520 bytes 2024-09-15T00:49:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 977945600 bytes 2024-09-15T00:49:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1025460224 bytes 2024-09-15T00:49:41.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1080973312 bytes 2024-09-15T00:49:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1133513728 bytes 2024-09-15T00:49:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1162254336 bytes 2024-09-15T00:49:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1219881984 bytes 2024-09-15T00:49:41.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1258324992 bytes 2024-09-15T00:49:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1280852992 bytes 2024-09-15T00:49:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1281456128 bytes 2024-09-15T00:49:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1308071936 bytes 2024-09-15T00:49:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1316071424 bytes 2024-09-15T00:49:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1339236352 bytes 2024-09-15T00:49:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1421768704 bytes 2024-09-15T00:49:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1463820288 bytes 2024-09-15T00:49:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1490682880 bytes 2024-09-15T00:49:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1537633280 bytes 2024-09-15T00:49:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1618702336 bytes 2024-09-15T00:49:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1651262464 bytes 2024-09-15T00:49:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1697202176 bytes 2024-09-15T00:49:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1725220864 bytes 2024-09-15T00:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1738438656 bytes 2024-09-15T00:49:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1790407680 bytes 2024-09-15T00:49:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1826085888 bytes 2024-09-15T00:49:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1873604608 bytes 2024-09-15T00:49:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1951698944 bytes 2024-09-15T00:49:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1978711040 bytes 2024-09-15T00:49:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2019107840 bytes 2024-09-15T00:49:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2028981248 bytes 2024-09-15T00:49:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2072612864 bytes 2024-09-15T00:49:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2079968256 bytes 2024-09-15T00:49:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2111587328 bytes 2024-09-15T00:49:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2127545344 bytes 2024-09-15T00:49:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2157598720 bytes 2024-09-15T00:49:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2215611392 bytes 2024-09-15T00:49:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2241338368 bytes 2024-09-15T00:49:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2317280256 bytes 2024-09-15T00:49:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2322796544 bytes 2024-09-15T00:49:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2357829632 bytes 2024-09-15T00:49:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2379481088 bytes 2024-09-15T00:49:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2428407808 bytes 2024-09-15T00:49:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2473833472 bytes 2024-09-15T00:49:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2491241472 bytes 2024-09-15T00:49:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2516475904 bytes 2024-09-15T00:49:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2541744128 bytes 2024-09-15T00:49:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2599313408 bytes 2024-09-15T00:49:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2657835008 bytes 2024-09-15T00:49:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2700722176 bytes 2024-09-15T00:49:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2757466112 bytes 2024-09-15T00:49:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2868228096 bytes 2024-09-15T00:49:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2928961536 bytes 2024-09-15T00:49:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2935058432 bytes 2024-09-15T00:49:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2948009984 bytes 2024-09-15T00:49:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2969742336 bytes 2024-09-15T00:49:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2991801344 bytes 2024-09-15T00:49:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3051468800 bytes 2024-09-15T00:49:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3142068224 bytes 2024-09-15T00:49:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3174432768 bytes 2024-09-15T00:49:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3175017472 bytes 2024-09-15T00:49:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3230690304 bytes 2024-09-15T00:49:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3296636928 bytes 2024-09-15T00:49:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3321547776 bytes 2024-09-15T00:49:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3348546560 bytes 2024-09-15T00:49:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3435972608 bytes 2024-09-15T00:49:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3510841344 bytes 2024-09-15T00:49:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3578368000 bytes 2024-09-15T00:49:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3645264896 bytes 2024-09-15T00:49:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3645881344 bytes 2024-09-15T00:49:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3678098432 bytes 2024-09-15T00:49:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3702902784 bytes 2024-09-15T00:49:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3765601280 bytes 2024-09-15T00:49:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3801177088 bytes 2024-09-15T00:49:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_19 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:49:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:49:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:49:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:49:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:49:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2024-09-15T00:49:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:49:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:49:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:49:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-15T00:49:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:49:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:49:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-15T00:49:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:49:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-15T00:49:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_19': 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1359265c40eb 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1359265c40eb 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:49:49 2024 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:49:49 2024 2024-09-15T00:49:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:49:49 2024 2024-09-15T00:49:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:49:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:49:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 237a420c-4d63-4519-a90b-522ac9d2b086 2024-09-15T00:49:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:49:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:49:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-15T00:49:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:49:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:49:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:49:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:49:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=13 2024-09-15T00:49:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:49:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:49:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 48769024 bytes 2024-09-15T00:49:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 89143296 bytes 2024-09-15T00:49:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 135816192 bytes 2024-09-15T00:49:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 168935424 bytes 2024-09-15T00:49:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 218582016 bytes 2024-09-15T00:49:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 248409088 bytes 2024-09-15T00:49:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 255630336 bytes 2024-09-15T00:49:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 279415808 bytes 2024-09-15T00:49:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 280004608 bytes 2024-09-15T00:49:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 338801664 bytes 2024-09-15T00:49:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 366479360 bytes 2024-09-15T00:49:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 413443072 bytes 2024-09-15T00:49:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 445761536 bytes 2024-09-15T00:49:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 523875328 bytes 2024-09-15T00:49:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 564730880 bytes 2024-09-15T00:49:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 656013312 bytes 2024-09-15T00:49:52.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 662552576 bytes 2024-09-15T00:49:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 765712384 bytes 2024-09-15T00:49:53.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 790791168 bytes 2024-09-15T00:49:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 824588288 bytes 2024-09-15T00:49:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 875333632 bytes 2024-09-15T00:49:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 954606592 bytes 2024-09-15T00:49:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1034451968 bytes 2024-09-15T00:49:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1100926976 bytes 2024-09-15T00:49:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1176634368 bytes 2024-09-15T00:49:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1198131200 bytes 2024-09-15T00:49:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1226619904 bytes 2024-09-15T00:49:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1270624256 bytes 2024-09-15T00:49:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1314289664 bytes 2024-09-15T00:49:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1379736576 bytes 2024-09-15T00:49:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1403272192 bytes 2024-09-15T00:49:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1443816448 bytes 2024-09-15T00:49:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1538530304 bytes 2024-09-15T00:49:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1565704192 bytes 2024-09-15T00:49:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1590435840 bytes 2024-09-15T00:49:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1637226496 bytes 2024-09-15T00:49:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1674294272 bytes 2024-09-15T00:49:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1736279040 bytes 2024-09-15T00:49:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1813567488 bytes 2024-09-15T00:49:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1835249664 bytes 2024-09-15T00:49:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1835865088 bytes 2024-09-15T00:49:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1846866944 bytes 2024-09-15T00:49:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1876130816 bytes 2024-09-15T00:49:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1944272896 bytes 2024-09-15T00:49:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1963725824 bytes 2024-09-15T00:49:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1985861632 bytes 2024-09-15T00:49:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2040409088 bytes 2024-09-15T00:49:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2083659776 bytes 2024-09-15T00:49:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2202497024 bytes 2024-09-15T00:49:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2223119360 bytes 2024-09-15T00:49:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2262369280 bytes 2024-09-15T00:49:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2324845568 bytes 2024-09-15T00:49:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2325435392 bytes 2024-09-15T00:49:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2362773504 bytes 2024-09-15T00:49:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2403414016 bytes 2024-09-15T00:49:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2426595328 bytes 2024-09-15T00:49:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2465831936 bytes 2024-09-15T00:49:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2467923968 bytes 2024-09-15T00:49:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2519218176 bytes 2024-09-15T00:49:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2545695744 bytes 2024-09-15T00:49:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2548099072 bytes 2024-09-15T00:49:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2575232000 bytes 2024-09-15T00:49:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2624189440 bytes 2024-09-15T00:49:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2624784384 bytes 2024-09-15T00:49:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2679854080 bytes 2024-09-15T00:49:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2691702784 bytes 2024-09-15T00:49:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2716089344 bytes 2024-09-15T00:49:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2754326528 bytes 2024-09-15T00:49:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2777204736 bytes 2024-09-15T00:49:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2802179072 bytes 2024-09-15T00:49:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2848641024 bytes 2024-09-15T00:49:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2849274880 bytes 2024-09-15T00:49:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2869492736 bytes 2024-09-15T00:49:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2879025152 bytes 2024-09-15T00:49:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2879660032 bytes 2024-09-15T00:49:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2880287744 bytes 2024-09-15T00:49:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 2956432384 bytes 2024-09-15T00:50:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 2994675712 bytes 2024-09-15T00:50:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3049412608 bytes 2024-09-15T00:50:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3166892032 bytes 2024-09-15T00:50:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3198164992 bytes 2024-09-15T00:50:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3210087424 bytes 2024-09-15T00:50:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3263869952 bytes 2024-09-15T00:50:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3332929536 bytes 2024-09-15T00:50:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3393955840 bytes 2024-09-15T00:50:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3473708032 bytes 2024-09-15T00:50:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3528164352 bytes 2024-09-15T00:50:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3592232960 bytes 2024-09-15T00:50:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3621234688 bytes 2024-09-15T00:50:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3697482752 bytes 2024-09-15T00:50:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3761965056 bytes 2024-09-15T00:50:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3800123392 bytes 2024-09-15T00:50:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3862677504 bytes 2024-09-15T00:50:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3877865472 bytes 2024-09-15T00:50:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 3925360640 bytes 2024-09-15T00:50:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4063352832 bytes 2024-09-15T00:50:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4105953280 bytes 2024-09-15T00:50:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4124999680 bytes 2024-09-15T00:50:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4147613696 bytes 2024-09-15T00:50:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4194423808 bytes 2024-09-15T00:50:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4215369728 bytes 2024-09-15T00:50:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4243003392 bytes 2024-09-15T00:50:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4256736256 bytes 2024-09-15T00:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_20 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-15T00:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2024-09-15T00:50:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:50:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:50:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-15T00:50:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:50:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-15T00:50:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:50:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-15T00:50:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_20': 2024-09-15T00:50:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:50:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:50:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:50:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13651b1da59 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13651b1da59 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:03 2024 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:03 2024 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:03 2024 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: ab51885a-03a7-4691-9f70-c8ead112d259 2024-09-15T00:50:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:50:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:50:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-15T00:50:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:50:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:50:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=9 2024-09-15T00:50:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:50:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:50:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 24003584 bytes 2024-09-15T00:50:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 119938048 bytes 2024-09-15T00:50:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 167363584 bytes 2024-09-15T00:50:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 211125248 bytes 2024-09-15T00:50:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 284851200 bytes 2024-09-15T00:50:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 300351488 bytes 2024-09-15T00:50:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 343821312 bytes 2024-09-15T00:50:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 386988032 bytes 2024-09-15T00:50:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 387593216 bytes 2024-09-15T00:50:05.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 388216832 bytes 2024-09-15T00:50:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 437102592 bytes 2024-09-15T00:50:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 459261952 bytes 2024-09-15T00:50:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 474552320 bytes 2024-09-15T00:50:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 507416576 bytes 2024-09-15T00:50:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 557186048 bytes 2024-09-15T00:50:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 611539968 bytes 2024-09-15T00:50:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 686354432 bytes 2024-09-15T00:50:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 753050624 bytes 2024-09-15T00:50:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 798013440 bytes 2024-09-15T00:50:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 799433728 bytes 2024-09-15T00:50:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 840376320 bytes 2024-09-15T00:50:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 871311360 bytes 2024-09-15T00:50:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 951764992 bytes 2024-09-15T00:50:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 986779648 bytes 2024-09-15T00:50:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1042623488 bytes 2024-09-15T00:50:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1072141312 bytes 2024-09-15T00:50:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1118007296 bytes 2024-09-15T00:50:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1167653888 bytes 2024-09-15T00:50:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1223909376 bytes 2024-09-15T00:50:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1290799104 bytes 2024-09-15T00:50:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1376347136 bytes 2024-09-15T00:50:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1429089280 bytes 2024-09-15T00:50:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1463302144 bytes 2024-09-15T00:50:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1479314432 bytes 2024-09-15T00:50:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1537269760 bytes 2024-09-15T00:50:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1591473152 bytes 2024-09-15T00:50:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1674528768 bytes 2024-09-15T00:50:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1767093248 bytes 2024-09-15T00:50:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1781273600 bytes 2024-09-15T00:50:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1818194944 bytes 2024-09-15T00:50:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1918703616 bytes 2024-09-15T00:50:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1961904128 bytes 2024-09-15T00:50:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2015606784 bytes 2024-09-15T00:50:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2016206848 bytes 2024-09-15T00:50:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2066293760 bytes 2024-09-15T00:50:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2088753152 bytes 2024-09-15T00:50:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2133391360 bytes 2024-09-15T00:50:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2163306496 bytes 2024-09-15T00:50:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2181521408 bytes 2024-09-15T00:50:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2221674496 bytes 2024-09-15T00:50:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2242362368 bytes 2024-09-15T00:50:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2259444736 bytes 2024-09-15T00:50:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2300434432 bytes 2024-09-15T00:50:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2308879360 bytes 2024-09-15T00:50:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2313012224 bytes 2024-09-15T00:50:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2362836992 bytes 2024-09-15T00:50:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2363430912 bytes 2024-09-15T00:50:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2399779840 bytes 2024-09-15T00:50:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2470932480 bytes 2024-09-15T00:50:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2487285760 bytes 2024-09-15T00:50:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2519563264 bytes 2024-09-15T00:50:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2560726016 bytes 2024-09-15T00:50:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2622398464 bytes 2024-09-15T00:50:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2693473280 bytes 2024-09-15T00:50:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2717110272 bytes 2024-09-15T00:50:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2759073792 bytes 2024-09-15T00:50:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2776726528 bytes 2024-09-15T00:50:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2820594688 bytes 2024-09-15T00:50:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2855138304 bytes 2024-09-15T00:50:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2877386752 bytes 2024-09-15T00:50:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3006723072 bytes 2024-09-15T00:50:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3045668864 bytes 2024-09-15T00:50:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3102362624 bytes 2024-09-15T00:50:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3127442432 bytes 2024-09-15T00:50:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:50:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:50:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:50:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:50:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_21 2024-09-15T00:50:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-15T00:50:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:50:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:50:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:50:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:50:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2024-09-15T00:50:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:50:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:50:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-15T00:50:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:50:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-15T00:50:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:50:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-15T00:50:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_21': 2024-09-15T00:50:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:50:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:50:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:50:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13714e1f1 2024-09-15T00:50:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13714e1f1 2024-09-15T00:50:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:50:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:50:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:50:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:50:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:13 2024 2024-09-15T00:50:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:13 2024 2024-09-15T00:50:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:13 2024 2024-09-15T00:50:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:50:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:50:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 69c7326f-a13b-465e-a7c0-8dec344e635a 2024-09-15T00:50:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:50:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:50:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-15T00:50:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:50:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:50:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=9 2024-09-15T00:50:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:50:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:50:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 23876608 bytes 2024-09-15T00:50:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 81541120 bytes 2024-09-15T00:50:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 190637056 bytes 2024-09-15T00:50:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 238006272 bytes 2024-09-15T00:50:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 262013952 bytes 2024-09-15T00:50:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 366027776 bytes 2024-09-15T00:50:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 418525184 bytes 2024-09-15T00:50:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 457664512 bytes 2024-09-15T00:50:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 460384256 bytes 2024-09-15T00:50:16.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.27402578859298654) > weight (0.25) 2024-09-15T00:50:16.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3972745259059075) > weight (0.25) 2024-09-15T00:50:16.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3036938804995878) > weight (0.25) 2024-09-15T00:50:16.045 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9061376331536535) > weight (0.25) 2024-09-15T00:50:16.045 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-15T00:50:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 478941184 bytes 2024-09-15T00:50:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 578803712 bytes 2024-09-15T00:50:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 672025600 bytes 2024-09-15T00:50:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 709184512 bytes 2024-09-15T00:50:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 776595456 bytes 2024-09-15T00:50:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 794598400 bytes 2024-09-15T00:50:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 819333120 bytes 2024-09-15T00:50:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 831622144 bytes 2024-09-15T00:50:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 853127168 bytes 2024-09-15T00:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 883034112 bytes 2024-09-15T00:50:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 903775232 bytes 2024-09-15T00:50:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 997981184 bytes 2024-09-15T00:50:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1016846336 bytes 2024-09-15T00:50:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1067518976 bytes 2024-09-15T00:50:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1092918272 bytes 2024-09-15T00:50:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1112028160 bytes 2024-09-15T00:50:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1172319232 bytes 2024-09-15T00:50:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1214881792 bytes 2024-09-15T00:50:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1215507456 bytes 2024-09-15T00:50:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1293395968 bytes 2024-09-15T00:50:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1314411520 bytes 2024-09-15T00:50:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1365095424 bytes 2024-09-15T00:50:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1401996288 bytes 2024-09-15T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1452018688 bytes 2024-09-15T00:50:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1465853952 bytes 2024-09-15T00:50:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1522464768 bytes 2024-09-15T00:50:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1523076096 bytes 2024-09-15T00:50:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1585457152 bytes 2024-09-15T00:50:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1623797760 bytes 2024-09-15T00:50:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1655318528 bytes 2024-09-15T00:50:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1664488448 bytes 2024-09-15T00:50:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1752811520 bytes 2024-09-15T00:50:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1777499136 bytes 2024-09-15T00:50:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1822926848 bytes 2024-09-15T00:50:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1880271872 bytes 2024-09-15T00:50:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1907278848 bytes 2024-09-15T00:50:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1924276224 bytes 2024-09-15T00:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1945644032 bytes 2024-09-15T00:50:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1948399616 bytes 2024-09-15T00:50:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2014008320 bytes 2024-09-15T00:50:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2112792576 bytes 2024-09-15T00:50:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2151814144 bytes 2024-09-15T00:50:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2163757056 bytes 2024-09-15T00:50:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2198327296 bytes 2024-09-15T00:50:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2275246080 bytes 2024-09-15T00:50:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2320012288 bytes 2024-09-15T00:50:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2361849856 bytes 2024-09-15T00:50:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2401533952 bytes 2024-09-15T00:50:21.830 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4706107638430673) > weight (0.25) 2024-09-15T00:50:21.830 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.421298899124835) > weight (0.25) 2024-09-15T00:50:21.830 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3721018893919451) > weight (0.25) 2024-09-15T00:50:21.830 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7027239009207272) > weight (0.25) 2024-09-15T00:50:21.830 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-09-15T00:50:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2443437056 bytes 2024-09-15T00:50:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2472288256 bytes 2024-09-15T00:50:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2498752512 bytes 2024-09-15T00:50:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2513944576 bytes 2024-09-15T00:50:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2551027712 bytes 2024-09-15T00:50:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2580125696 bytes 2024-09-15T00:50:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2580745216 bytes 2024-09-15T00:50:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2668033024 bytes 2024-09-15T00:50:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2734075904 bytes 2024-09-15T00:50:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2799152128 bytes 2024-09-15T00:50:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2800150528 bytes 2024-09-15T00:50:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2875979776 bytes 2024-09-15T00:50:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2917596160 bytes 2024-09-15T00:50:23.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3009330176 bytes 2024-09-15T00:50:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3032271872 bytes 2024-09-15T00:50:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_22 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:50:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-15T00:50:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:50:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:50:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:50:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:50:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2024-09-15T00:50:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:50:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:50:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-15T00:50:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:50:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-15T00:50:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:50:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-15T00:50:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_22': 2024-09-15T00:50:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:50:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:50:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:50:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 137dcd19560e 2024-09-15T00:50:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.137dcd19560e 2024-09-15T00:50:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:50:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:50:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:50:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:50:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:23 2024 2024-09-15T00:50:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:23 2024 2024-09-15T00:50:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:23 2024 2024-09-15T00:50:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:50:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:50:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 0559d020-e414-4263-981e-562438312202 2024-09-15T00:50:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:50:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:50:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-15T00:50:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:50:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:50:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=6 2024-09-15T00:50:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:50:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:50:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 23725056 bytes 2024-09-15T00:50:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 155642880 bytes 2024-09-15T00:50:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 217322496 bytes 2024-09-15T00:50:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 252542976 bytes 2024-09-15T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 299394048 bytes 2024-09-15T00:50:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 322332672 bytes 2024-09-15T00:50:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 338146304 bytes 2024-09-15T00:50:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 356340736 bytes 2024-09-15T00:50:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 388495360 bytes 2024-09-15T00:50:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 402844672 bytes 2024-09-15T00:50:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 463911936 bytes 2024-09-15T00:50:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 508810240 bytes 2024-09-15T00:50:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 579146752 bytes 2024-09-15T00:50:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 649923584 bytes 2024-09-15T00:50:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 674544640 bytes 2024-09-15T00:50:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 714124288 bytes 2024-09-15T00:50:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 749544448 bytes 2024-09-15T00:50:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 792233984 bytes 2024-09-15T00:50:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 814455808 bytes 2024-09-15T00:50:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 836127744 bytes 2024-09-15T00:50:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 861242368 bytes 2024-09-15T00:50:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 906264576 bytes 2024-09-15T00:50:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 980748288 bytes 2024-09-15T00:50:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1004589056 bytes 2024-09-15T00:50:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1030729728 bytes 2024-09-15T00:50:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1082467328 bytes 2024-09-15T00:50:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1113393152 bytes 2024-09-15T00:50:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1119633408 bytes 2024-09-15T00:50:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1120245760 bytes 2024-09-15T00:50:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1168032768 bytes 2024-09-15T00:50:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1214295040 bytes 2024-09-15T00:50:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1233569792 bytes 2024-09-15T00:50:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1247768576 bytes 2024-09-15T00:50:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1304919040 bytes 2024-09-15T00:50:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1369160704 bytes 2024-09-15T00:50:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1429945344 bytes 2024-09-15T00:50:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1459928064 bytes 2024-09-15T00:50:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1533294592 bytes 2024-09-15T00:50:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1554126848 bytes 2024-09-15T00:50:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1559094272 bytes 2024-09-15T00:50:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1602466816 bytes 2024-09-15T00:50:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1633072128 bytes 2024-09-15T00:50:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1692272640 bytes 2024-09-15T00:50:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1796776960 bytes 2024-09-15T00:50:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1810226176 bytes 2024-09-15T00:50:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1819313152 bytes 2024-09-15T00:50:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1827363840 bytes 2024-09-15T00:50:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:50:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:50:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:50:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:50:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_23 2024-09-15T00:50:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-15T00:50:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:50:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:50:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-15T00:50:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2024-09-15T00:50:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:50:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:50:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-15T00:50:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:50:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-15T00:50:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:50:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-15T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_23': 2024-09-15T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1389af95a75e 2024-09-15T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1389af95a75e 2024-09-15T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:50:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:50:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:30 2024 2024-09-15T00:50:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:30 2024 2024-09-15T00:50:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:30 2024 2024-09-15T00:50:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:50:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:50:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 80ac0c68-49b7-4ebc-8de0-5aae313e243b 2024-09-15T00:50:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-15T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:50:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=12 2024-09-15T00:50:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:50:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:50:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 50808832 bytes 2024-09-15T00:50:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 97766400 bytes 2024-09-15T00:50:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 99456000 bytes 2024-09-15T00:50:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 183797760 bytes 2024-09-15T00:50:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 218465280 bytes 2024-09-15T00:50:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 252745728 bytes 2024-09-15T00:50:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 257745920 bytes 2024-09-15T00:50:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 370365440 bytes 2024-09-15T00:50:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 436889600 bytes 2024-09-15T00:50:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 437523456 bytes 2024-09-15T00:50:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 509304832 bytes 2024-09-15T00:50:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 554227712 bytes 2024-09-15T00:50:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 612725760 bytes 2024-09-15T00:50:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 654151680 bytes 2024-09-15T00:50:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 666400768 bytes 2024-09-15T00:50:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 706606080 bytes 2024-09-15T00:50:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 731831296 bytes 2024-09-15T00:50:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 756591616 bytes 2024-09-15T00:50:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 769270784 bytes 2024-09-15T00:50:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 780408832 bytes 2024-09-15T00:50:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 853352448 bytes 2024-09-15T00:50:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 862113792 bytes 2024-09-15T00:50:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 915119104 bytes 2024-09-15T00:50:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 940821504 bytes 2024-09-15T00:50:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 999834624 bytes 2024-09-15T00:50:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1073045504 bytes 2024-09-15T00:50:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1095047168 bytes 2024-09-15T00:50:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1102967808 bytes 2024-09-15T00:50:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1147657216 bytes 2024-09-15T00:50:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1190441984 bytes 2024-09-15T00:50:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1259270144 bytes 2024-09-15T00:50:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1339487232 bytes 2024-09-15T00:50:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1378172928 bytes 2024-09-15T00:50:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1440333824 bytes 2024-09-15T00:50:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1457062912 bytes 2024-09-15T00:50:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1527694336 bytes 2024-09-15T00:50:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1629421568 bytes 2024-09-15T00:50:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1647114240 bytes 2024-09-15T00:50:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1686340608 bytes 2024-09-15T00:50:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1701098496 bytes 2024-09-15T00:50:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1747385344 bytes 2024-09-15T00:50:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1765909504 bytes 2024-09-15T00:50:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1854700544 bytes 2024-09-15T00:50:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1895161856 bytes 2024-09-15T00:50:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1986802688 bytes 2024-09-15T00:50:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2003581952 bytes 2024-09-15T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2044244992 bytes 2024-09-15T00:50:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2096189440 bytes 2024-09-15T00:50:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2126568448 bytes 2024-09-15T00:50:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2158385152 bytes 2024-09-15T00:50:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2219277312 bytes 2024-09-15T00:50:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2275511296 bytes 2024-09-15T00:50:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2322732032 bytes 2024-09-15T00:50:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2363010048 bytes 2024-09-15T00:50:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2437441536 bytes 2024-09-15T00:50:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2457578496 bytes 2024-09-15T00:50:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2515574784 bytes 2024-09-15T00:50:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2535237632 bytes 2024-09-15T00:50:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2591894528 bytes 2024-09-15T00:50:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2606627840 bytes 2024-09-15T00:50:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2669251584 bytes 2024-09-15T00:50:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2748879872 bytes 2024-09-15T00:50:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2793711616 bytes 2024-09-15T00:50:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2822980608 bytes 2024-09-15T00:50:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2847602688 bytes 2024-09-15T00:50:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2880575488 bytes 2024-09-15T00:50:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2898156544 bytes 2024-09-15T00:50:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2962055168 bytes 2024-09-15T00:50:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2997138432 bytes 2024-09-15T00:50:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3047974912 bytes 2024-09-15T00:50:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3122872320 bytes 2024-09-15T00:50:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3156118528 bytes 2024-09-15T00:50:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3163104256 bytes 2024-09-15T00:50:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3192190976 bytes 2024-09-15T00:50:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3220186112 bytes 2024-09-15T00:50:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3283676160 bytes 2024-09-15T00:50:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3294216192 bytes 2024-09-15T00:50:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3356008448 bytes 2024-09-15T00:50:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3400423424 bytes 2024-09-15T00:50:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3455651840 bytes 2024-09-15T00:50:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3481210880 bytes 2024-09-15T00:50:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3515331584 bytes 2024-09-15T00:50:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3579096064 bytes 2024-09-15T00:50:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3621663744 bytes 2024-09-15T00:50:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3677285376 bytes 2024-09-15T00:50:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3764530176 bytes 2024-09-15T00:50:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3791664128 bytes 2024-09-15T00:50:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3849992192 bytes 2024-09-15T00:50:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3925477376 bytes 2024-09-15T00:50:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3982210048 bytes 2024-09-15T00:50:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 4015114240 bytes 2024-09-15T00:50:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 4047025152 bytes 2024-09-15T00:50:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 4057252864 bytes 2024-09-15T00:50:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4072765440 bytes 2024-09-15T00:50:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4099360768 bytes 2024-09-15T00:50:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:50:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:50:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_24 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2024-09-15T00:50:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:50:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:50:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-15T00:50:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:50:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-15T00:50:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:50:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-15T00:50:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_24': 2024-09-15T00:50:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:50:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:50:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:50:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13958c0791c3 2024-09-15T00:50:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13958c0791c3 2024-09-15T00:50:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:50:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:50:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:50:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:50:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:43 2024 2024-09-15T00:50:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:43 2024 2024-09-15T00:50:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:43 2024 2024-09-15T00:50:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:50:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:50:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: a26c1dea-71ff-4c4c-ba15-84859056bc8f 2024-09-15T00:50:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:50:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:50:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-15T00:50:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:50:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:50:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=13 2024-09-15T00:50:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:50:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:50:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 19289088 bytes 2024-09-15T00:50:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 77624320 bytes 2024-09-15T00:50:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 82092032 bytes 2024-09-15T00:50:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 85304320 bytes 2024-09-15T00:50:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 135392256 bytes 2024-09-15T00:50:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 207980544 bytes 2024-09-15T00:50:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 283190272 bytes 2024-09-15T00:50:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 325511168 bytes 2024-09-15T00:50:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 433512448 bytes 2024-09-15T00:50:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 498077696 bytes 2024-09-15T00:50:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 588525568 bytes 2024-09-15T00:50:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 623453184 bytes 2024-09-15T00:50:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 678721536 bytes 2024-09-15T00:50:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 726785024 bytes 2024-09-15T00:50:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 737659904 bytes 2024-09-15T00:50:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 790971392 bytes 2024-09-15T00:50:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 832357376 bytes 2024-09-15T00:50:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 852368384 bytes 2024-09-15T00:50:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 905100288 bytes 2024-09-15T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 920018944 bytes 2024-09-15T00:50:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 969610240 bytes 2024-09-15T00:50:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1000886272 bytes 2024-09-15T00:50:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1043462144 bytes 2024-09-15T00:50:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1103489024 bytes 2024-09-15T00:50:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1132291072 bytes 2024-09-15T00:50:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1174302720 bytes 2024-09-15T00:50:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1207142400 bytes 2024-09-15T00:50:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1278526464 bytes 2024-09-15T00:50:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1312780288 bytes 2024-09-15T00:50:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1358221312 bytes 2024-09-15T00:50:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1371678720 bytes 2024-09-15T00:50:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1378196480 bytes 2024-09-15T00:50:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1418241024 bytes 2024-09-15T00:50:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1459552256 bytes 2024-09-15T00:50:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1460176896 bytes 2024-09-15T00:50:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1501035520 bytes 2024-09-15T00:50:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1524896768 bytes 2024-09-15T00:50:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1561455616 bytes 2024-09-15T00:50:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1617036288 bytes 2024-09-15T00:50:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1657632768 bytes 2024-09-15T00:50:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1679872000 bytes 2024-09-15T00:50:50.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1685498880 bytes 2024-09-15T00:50:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1699171328 bytes 2024-09-15T00:50:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1720209408 bytes 2024-09-15T00:50:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1753391104 bytes 2024-09-15T00:50:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1790849024 bytes 2024-09-15T00:50:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1827501056 bytes 2024-09-15T00:50:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1828086784 bytes 2024-09-15T00:50:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1852745728 bytes 2024-09-15T00:50:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1940749312 bytes 2024-09-15T00:50:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1941382144 bytes 2024-09-15T00:50:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 1941990400 bytes 2024-09-15T00:50:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 1947938816 bytes 2024-09-15T00:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 1965011968 bytes 2024-09-15T00:50:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2037580800 bytes 2024-09-15T00:50:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2039981056 bytes 2024-09-15T00:50:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2057915392 bytes 2024-09-15T00:50:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2076725248 bytes 2024-09-15T00:50:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2100732928 bytes 2024-09-15T00:50:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2134586368 bytes 2024-09-15T00:50:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2138620928 bytes 2024-09-15T00:50:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2206652416 bytes 2024-09-15T00:50:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2285379584 bytes 2024-09-15T00:50:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2289513472 bytes 2024-09-15T00:50:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2312346624 bytes 2024-09-15T00:50:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2333202432 bytes 2024-09-15T00:50:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2423851008 bytes 2024-09-15T00:50:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2436913152 bytes 2024-09-15T00:50:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2438157312 bytes 2024-09-15T00:50:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2467528704 bytes 2024-09-15T00:50:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2514350080 bytes 2024-09-15T00:50:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2535014400 bytes 2024-09-15T00:50:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2557663232 bytes 2024-09-15T00:50:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2586219520 bytes 2024-09-15T00:50:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2683266048 bytes 2024-09-15T00:50:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2750840832 bytes 2024-09-15T00:50:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 2811640832 bytes 2024-09-15T00:50:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 2861631488 bytes 2024-09-15T00:50:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 2916267008 bytes 2024-09-15T00:50:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 2924101632 bytes 2024-09-15T00:50:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 2992480256 bytes 2024-09-15T00:50:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3040339968 bytes 2024-09-15T00:50:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3072499712 bytes 2024-09-15T00:50:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3108245504 bytes 2024-09-15T00:50:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3121525760 bytes 2024-09-15T00:50:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3186023424 bytes 2024-09-15T00:50:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3249466368 bytes 2024-09-15T00:50:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3266900992 bytes 2024-09-15T00:50:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3297732608 bytes 2024-09-15T00:50:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3320929280 bytes 2024-09-15T00:50:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3388778496 bytes 2024-09-15T00:50:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3421746176 bytes 2024-09-15T00:50:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3441329152 bytes 2024-09-15T00:50:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3507938304 bytes 2024-09-15T00:50:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 3629038592 bytes 2024-09-15T00:50:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 3667632128 bytes 2024-09-15T00:50:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 3674708992 bytes 2024-09-15T00:50:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 3691699200 bytes 2024-09-15T00:50:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 3736921088 bytes 2024-09-15T00:50:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 3799213056 bytes 2024-09-15T00:50:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 3821276160 bytes 2024-09-15T00:50:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 3848550400 bytes 2024-09-15T00:50:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 3907992576 bytes 2024-09-15T00:50:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 3942066176 bytes 2024-09-15T00:50:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 3994202112 bytes 2024-09-15T00:50:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 4026162176 bytes 2024-09-15T00:50:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4075882496 bytes 2024-09-15T00:50:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:50:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:50:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:50:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:50:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_25 2024-09-15T00:50:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-15T00:50:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:50:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:50:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-15T00:50:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:50:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:50:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:50:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:50:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:50:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2024-09-15T00:50:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:50:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:50:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:50:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-15T00:50:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:50:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:50:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-15T00:50:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:50:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-15T00:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_25': 2024-09-15T00:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13a118ce80a0 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13a118ce80a0 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:57 2024 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:57 2024 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:57 2024 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 8c25e489-380c-471b-b67e-670250240dc3 2024-09-15T00:50:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:50:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:50:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-15T00:50:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:50:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:50:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:50:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:50:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=12 2024-09-15T00:50:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:50:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:50:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 56750080 bytes 2024-09-15T00:50:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 98589696 bytes 2024-09-15T00:50:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 220978176 bytes 2024-09-15T00:50:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 238040064 bytes 2024-09-15T00:50:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 252412928 bytes 2024-09-15T00:50:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 291494912 bytes 2024-09-15T00:50:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 300472320 bytes 2024-09-15T00:51:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 352222208 bytes 2024-09-15T00:51:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 376739840 bytes 2024-09-15T00:51:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 398482432 bytes 2024-09-15T00:51:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 456548352 bytes 2024-09-15T00:51:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 536966144 bytes 2024-09-15T00:51:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 587791360 bytes 2024-09-15T00:51:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 658803712 bytes 2024-09-15T00:51:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 677206016 bytes 2024-09-15T00:51:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 776025088 bytes 2024-09-15T00:51:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 837469184 bytes 2024-09-15T00:51:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 858484736 bytes 2024-09-15T00:51:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 871860224 bytes 2024-09-15T00:51:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 904909824 bytes 2024-09-15T00:51:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 924329984 bytes 2024-09-15T00:51:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 933279744 bytes 2024-09-15T00:51:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 952826880 bytes 2024-09-15T00:51:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1044686848 bytes 2024-09-15T00:51:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1055865856 bytes 2024-09-15T00:51:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1141647360 bytes 2024-09-15T00:51:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1174369280 bytes 2024-09-15T00:51:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1226216448 bytes 2024-09-15T00:51:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1263913984 bytes 2024-09-15T00:51:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1325426688 bytes 2024-09-15T00:51:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1333987328 bytes 2024-09-15T00:51:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1355348992 bytes 2024-09-15T00:51:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1384563712 bytes 2024-09-15T00:51:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1461006336 bytes 2024-09-15T00:51:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1487621120 bytes 2024-09-15T00:51:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1633190912 bytes 2024-09-15T00:51:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1645370368 bytes 2024-09-15T00:51:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1723145216 bytes 2024-09-15T00:51:04.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1785022464 bytes 2024-09-15T00:51:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1853289472 bytes 2024-09-15T00:51:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1887596544 bytes 2024-09-15T00:51:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1928115200 bytes 2024-09-15T00:51:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1932341248 bytes 2024-09-15T00:51:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1989915648 bytes 2024-09-15T00:51:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2026697728 bytes 2024-09-15T00:51:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2075274240 bytes 2024-09-15T00:51:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2149059584 bytes 2024-09-15T00:51:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2197197824 bytes 2024-09-15T00:51:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2258501632 bytes 2024-09-15T00:51:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2292791296 bytes 2024-09-15T00:51:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2352782336 bytes 2024-09-15T00:51:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2409442304 bytes 2024-09-15T00:51:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2465389568 bytes 2024-09-15T00:51:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2475116544 bytes 2024-09-15T00:51:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2532117504 bytes 2024-09-15T00:51:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2630116352 bytes 2024-09-15T00:51:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2680041472 bytes 2024-09-15T00:51:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2741420032 bytes 2024-09-15T00:51:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2811114496 bytes 2024-09-15T00:51:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2816078848 bytes 2024-09-15T00:51:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2901431296 bytes 2024-09-15T00:51:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2951027712 bytes 2024-09-15T00:51:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2974808064 bytes 2024-09-15T00:51:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 3030533120 bytes 2024-09-15T00:51:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 3098818560 bytes 2024-09-15T00:51:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 3202836480 bytes 2024-09-15T00:51:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3224223744 bytes 2024-09-15T00:51:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3238981632 bytes 2024-09-15T00:51:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3264424960 bytes 2024-09-15T00:51:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3313301504 bytes 2024-09-15T00:51:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3406905344 bytes 2024-09-15T00:51:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3464375296 bytes 2024-09-15T00:51:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3492365312 bytes 2024-09-15T00:51:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3529476096 bytes 2024-09-15T00:51:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3534218240 bytes 2024-09-15T00:51:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3626950656 bytes 2024-09-15T00:51:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3658162176 bytes 2024-09-15T00:51:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3726801920 bytes 2024-09-15T00:51:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3744468992 bytes 2024-09-15T00:51:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3789115392 bytes 2024-09-15T00:51:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3815392256 bytes 2024-09-15T00:51:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3829129216 bytes 2024-09-15T00:51:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3862656000 bytes 2024-09-15T00:51:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3943805952 bytes 2024-09-15T00:51:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3963704320 bytes 2024-09-15T00:51:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 4042194944 bytes 2024-09-15T00:51:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 4070332416 bytes 2024-09-15T00:51:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 4096322560 bytes 2024-09-15T00:51:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 4125077504 bytes 2024-09-15T00:51:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 4152270848 bytes 2024-09-15T00:51:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 4269923328 bytes 2024-09-15T00:51:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 4328525824 bytes 2024-09-15T00:51:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:51:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:51:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:51:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:51:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_26 2024-09-15T00:51:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:51:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:51:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:51:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:51:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2024-09-15T00:51:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:51:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:51:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-15T00:51:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:51:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-15T00:51:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:51:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-15T00:51:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_26': 2024-09-15T00:51:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:51:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:51:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:51:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13ad515f31af 2024-09-15T00:51:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13ad515f31af 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:11 2024 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:11 2024 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:11 2024 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 22c39bf0-0e09-4cf6-b50b-7c174804bc9e 2024-09-15T00:51:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:51:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:51:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-15T00:51:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:51:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:51:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=5 2024-09-15T00:51:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:51:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:51:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 45826048 bytes 2024-09-15T00:51:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 64193536 bytes 2024-09-15T00:51:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 144228352 bytes 2024-09-15T00:51:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 175774720 bytes 2024-09-15T00:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 264269824 bytes 2024-09-15T00:51:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 294072320 bytes 2024-09-15T00:51:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 313543680 bytes 2024-09-15T00:51:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 410535936 bytes 2024-09-15T00:51:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 435454976 bytes 2024-09-15T00:51:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 463332352 bytes 2024-09-15T00:51:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 489261056 bytes 2024-09-15T00:51:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 489881600 bytes 2024-09-15T00:51:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 526694400 bytes 2024-09-15T00:51:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 569791488 bytes 2024-09-15T00:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 582879232 bytes 2024-09-15T00:51:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 621737984 bytes 2024-09-15T00:51:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 648901632 bytes 2024-09-15T00:51:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 686673920 bytes 2024-09-15T00:51:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 761991168 bytes 2024-09-15T00:51:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 810760192 bytes 2024-09-15T00:51:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 828429312 bytes 2024-09-15T00:51:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 863685632 bytes 2024-09-15T00:51:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 887267328 bytes 2024-09-15T00:51:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 940131328 bytes 2024-09-15T00:51:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 962342912 bytes 2024-09-15T00:51:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1029409792 bytes 2024-09-15T00:51:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1085846528 bytes 2024-09-15T00:51:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1135248384 bytes 2024-09-15T00:51:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1185629184 bytes 2024-09-15T00:51:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1190632448 bytes 2024-09-15T00:51:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1212908544 bytes 2024-09-15T00:51:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1255153664 bytes 2024-09-15T00:51:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1282404352 bytes 2024-09-15T00:51:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1326402560 bytes 2024-09-15T00:51:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1341785088 bytes 2024-09-15T00:51:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1387774976 bytes 2024-09-15T00:51:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1407096832 bytes 2024-09-15T00:51:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1456761856 bytes 2024-09-15T00:51:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1503834112 bytes 2024-09-15T00:51:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1518921728 bytes 2024-09-15T00:51:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:51:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_27 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2024-09-15T00:51:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:51:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:51:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-15T00:51:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:51:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-15T00:51:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:51:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-15T00:51:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_27': 2024-09-15T00:51:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:51:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:51:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:51:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13b980bca3a1 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13b980bca3a1 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:16 2024 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:16 2024 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:16 2024 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 10250805-8ae3-4f07-92f0-130437cc82f8 2024-09-15T00:51:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:51:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:51:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-15T00:51:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:51:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=11 2024-09-15T00:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:51:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 40710144 bytes 2024-09-15T00:51:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 79887360 bytes 2024-09-15T00:51:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 116590592 bytes 2024-09-15T00:51:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 137378816 bytes 2024-09-15T00:51:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 172614656 bytes 2024-09-15T00:51:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 187318272 bytes 2024-09-15T00:51:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 240669696 bytes 2024-09-15T00:51:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 265229312 bytes 2024-09-15T00:51:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 304953344 bytes 2024-09-15T00:51:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 320980992 bytes 2024-09-15T00:51:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 386159616 bytes 2024-09-15T00:51:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 403791872 bytes 2024-09-15T00:51:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 437818368 bytes 2024-09-15T00:51:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 451789824 bytes 2024-09-15T00:51:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 485271552 bytes 2024-09-15T00:51:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 485869568 bytes 2024-09-15T00:51:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 521818112 bytes 2024-09-15T00:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 522438656 bytes 2024-09-15T00:51:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 588395520 bytes 2024-09-15T00:51:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 608452608 bytes 2024-09-15T00:51:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 655009792 bytes 2024-09-15T00:51:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 682480640 bytes 2024-09-15T00:51:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 706564096 bytes 2024-09-15T00:51:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 730703872 bytes 2024-09-15T00:51:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 778724352 bytes 2024-09-15T00:51:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 794000384 bytes 2024-09-15T00:51:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 827481088 bytes 2024-09-15T00:51:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 851238912 bytes 2024-09-15T00:51:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 902402048 bytes 2024-09-15T00:51:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 918730752 bytes 2024-09-15T00:51:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 932016128 bytes 2024-09-15T00:51:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1076784128 bytes 2024-09-15T00:51:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1082048512 bytes 2024-09-15T00:51:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1169544192 bytes 2024-09-15T00:51:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1243560960 bytes 2024-09-15T00:51:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1304244224 bytes 2024-09-15T00:51:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1329022976 bytes 2024-09-15T00:51:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1412936704 bytes 2024-09-15T00:51:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1430733824 bytes 2024-09-15T00:51:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1436681216 bytes 2024-09-15T00:51:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1451534336 bytes 2024-09-15T00:51:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1477906432 bytes 2024-09-15T00:51:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1530204160 bytes 2024-09-15T00:51:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1548220416 bytes 2024-09-15T00:51:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1596145664 bytes 2024-09-15T00:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1639535616 bytes 2024-09-15T00:51:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1691091968 bytes 2024-09-15T00:51:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1736402944 bytes 2024-09-15T00:51:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1785054208 bytes 2024-09-15T00:51:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1806727168 bytes 2024-09-15T00:51:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1807335424 bytes 2024-09-15T00:51:24.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 1884147712 bytes 2024-09-15T00:51:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 1956515840 bytes 2024-09-15T00:51:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 1998198784 bytes 2024-09-15T00:51:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2016076800 bytes 2024-09-15T00:51:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2055257088 bytes 2024-09-15T00:51:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2066351104 bytes 2024-09-15T00:51:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2101748736 bytes 2024-09-15T00:51:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2126360576 bytes 2024-09-15T00:51:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2147511296 bytes 2024-09-15T00:51:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2212446208 bytes 2024-09-15T00:51:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2251539456 bytes 2024-09-15T00:51:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2265629696 bytes 2024-09-15T00:51:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2299356160 bytes 2024-09-15T00:51:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2301924352 bytes 2024-09-15T00:51:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2312103936 bytes 2024-09-15T00:51:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2357976064 bytes 2024-09-15T00:51:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2400819200 bytes 2024-09-15T00:51:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2431705088 bytes 2024-09-15T00:51:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2462808064 bytes 2024-09-15T00:51:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2488922112 bytes 2024-09-15T00:51:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2550578176 bytes 2024-09-15T00:51:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2598104064 bytes 2024-09-15T00:51:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2622741504 bytes 2024-09-15T00:51:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2635062272 bytes 2024-09-15T00:51:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2670895104 bytes 2024-09-15T00:51:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 2698231808 bytes 2024-09-15T00:51:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 2738841600 bytes 2024-09-15T00:51:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 2750641152 bytes 2024-09-15T00:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 2799595520 bytes 2024-09-15T00:51:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 2851817472 bytes 2024-09-15T00:51:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 2852446208 bytes 2024-09-15T00:51:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 2923097088 bytes 2024-09-15T00:51:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 2978851840 bytes 2024-09-15T00:51:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 2990057472 bytes 2024-09-15T00:51:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3056607232 bytes 2024-09-15T00:51:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3079970816 bytes 2024-09-15T00:51:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3135443968 bytes 2024-09-15T00:51:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3158555648 bytes 2024-09-15T00:51:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3249510400 bytes 2024-09-15T00:51:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3325963264 bytes 2024-09-15T00:51:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3398287360 bytes 2024-09-15T00:51:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:51:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:51:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:51:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:51:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_28 2024-09-15T00:51:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-15T00:51:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:51:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:51:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:51:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2024-09-15T00:51:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:51:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:51:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-15T00:51:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:51:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-15T00:51:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:51:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-15T00:51:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_28': 2024-09-15T00:51:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:51:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:51:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:51:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13c5889c1079 2024-09-15T00:51:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13c5889c1079 2024-09-15T00:51:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:51:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:51:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:51:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:51:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:28 2024 2024-09-15T00:51:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:28 2024 2024-09-15T00:51:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:28 2024 2024-09-15T00:51:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:51:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:51:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 5806c03a-ef8d-41cb-80aa-38c353b9ac79 2024-09-15T00:51:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:51:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:51:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-15T00:51:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:51:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:51:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=5 2024-09-15T00:51:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:51:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:51:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 23333888 bytes 2024-09-15T00:51:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 27137024 bytes 2024-09-15T00:51:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 105938944 bytes 2024-09-15T00:51:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 106551296 bytes 2024-09-15T00:51:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 145419264 bytes 2024-09-15T00:51:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 249115648 bytes 2024-09-15T00:51:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 310910976 bytes 2024-09-15T00:51:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 311535616 bytes 2024-09-15T00:51:31.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 416355328 bytes 2024-09-15T00:51:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 502676480 bytes 2024-09-15T00:51:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 582883328 bytes 2024-09-15T00:51:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 600894464 bytes 2024-09-15T00:51:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 657368064 bytes 2024-09-15T00:51:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 727928832 bytes 2024-09-15T00:51:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 743988224 bytes 2024-09-15T00:51:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 785597440 bytes 2024-09-15T00:51:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 820768768 bytes 2024-09-15T00:51:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 854807552 bytes 2024-09-15T00:51:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 885525504 bytes 2024-09-15T00:51:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 894088192 bytes 2024-09-15T00:51:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 924829696 bytes 2024-09-15T00:51:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 930796544 bytes 2024-09-15T00:51:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 950356992 bytes 2024-09-15T00:51:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 974329856 bytes 2024-09-15T00:51:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1040654336 bytes 2024-09-15T00:51:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1041277952 bytes 2024-09-15T00:51:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1093342208 bytes 2024-09-15T00:51:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1122989056 bytes 2024-09-15T00:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1159393280 bytes 2024-09-15T00:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1198064640 bytes 2024-09-15T00:51:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1229537280 bytes 2024-09-15T00:51:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1323064320 bytes 2024-09-15T00:51:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1414365184 bytes 2024-09-15T00:51:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1474956288 bytes 2024-09-15T00:51:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1550363648 bytes 2024-09-15T00:51:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1611356160 bytes 2024-09-15T00:51:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1708428288 bytes 2024-09-15T00:51:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1753368576 bytes 2024-09-15T00:51:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:51:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:51:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:51:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_29 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:51:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:51:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:51:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:51:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2024-09-15T00:51:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:51:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:51:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-15T00:51:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:51:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-15T00:51:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:51:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_29': 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13d15c88b3f2 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13d15c88b3f2 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:51:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:34 2024 2024-09-15T00:51:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:34 2024 2024-09-15T00:51:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:34 2024 2024-09-15T00:51:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:51:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:51:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: b144ab42-9839-4e90-9820-f5ac9d0bf688 2024-09-15T00:51:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:51:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:51:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-15T00:51:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:51:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:51:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=6 2024-09-15T00:51:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:51:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:51:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 30440448 bytes 2024-09-15T00:51:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 75618304 bytes 2024-09-15T00:51:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 95522816 bytes 2024-09-15T00:51:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 118607872 bytes 2024-09-15T00:51:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 193152000 bytes 2024-09-15T00:51:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 234948608 bytes 2024-09-15T00:51:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 265906176 bytes 2024-09-15T00:51:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 321728512 bytes 2024-09-15T00:51:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 366018560 bytes 2024-09-15T00:51:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 367484928 bytes 2024-09-15T00:51:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 399361024 bytes 2024-09-15T00:51:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 425944064 bytes 2024-09-15T00:51:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 497880064 bytes 2024-09-15T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 519782400 bytes 2024-09-15T00:51:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 527672320 bytes 2024-09-15T00:51:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 552790016 bytes 2024-09-15T00:51:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 579914752 bytes 2024-09-15T00:51:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 610412544 bytes 2024-09-15T00:51:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 650881024 bytes 2024-09-15T00:51:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 673433600 bytes 2024-09-15T00:51:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 703531008 bytes 2024-09-15T00:51:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 745115648 bytes 2024-09-15T00:51:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 791770112 bytes 2024-09-15T00:51:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 792389632 bytes 2024-09-15T00:51:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 860191744 bytes 2024-09-15T00:51:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 958551040 bytes 2024-09-15T00:51:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 986391552 bytes 2024-09-15T00:51:39.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1050182656 bytes 2024-09-15T00:51:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1106145280 bytes 2024-09-15T00:51:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1215808512 bytes 2024-09-15T00:51:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1285265408 bytes 2024-09-15T00:51:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1342219264 bytes 2024-09-15T00:51:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1393225728 bytes 2024-09-15T00:51:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1402512384 bytes 2024-09-15T00:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1470284800 bytes 2024-09-15T00:51:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1495390208 bytes 2024-09-15T00:51:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1522584576 bytes 2024-09-15T00:51:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1523202048 bytes 2024-09-15T00:51:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1536327680 bytes 2024-09-15T00:51:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1548484608 bytes 2024-09-15T00:51:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1628043264 bytes 2024-09-15T00:51:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1642546176 bytes 2024-09-15T00:51:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1653183488 bytes 2024-09-15T00:51:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1653825536 bytes 2024-09-15T00:51:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1676394496 bytes 2024-09-15T00:51:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1725850624 bytes 2024-09-15T00:51:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1785462784 bytes 2024-09-15T00:51:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:51:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:51:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_30 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:51:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2024-09-15T00:51:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:51:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:51:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-15T00:51:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:51:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-15T00:51:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:51:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-15T00:51:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_30': 2024-09-15T00:51:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:51:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:51:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13dd9d223fc4 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13dd9d223fc4 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:42 2024 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:42 2024 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:42 2024 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 5efc243e-6384-4dc5-97d4-ee66a1a5f7ec 2024-09-15T00:51:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:51:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:51:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-15T00:51:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:51:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:51:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=6 2024-09-15T00:51:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:51:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:51:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 29970432 bytes 2024-09-15T00:51:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 94676992 bytes 2024-09-15T00:51:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 116417536 bytes 2024-09-15T00:51:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 143095808 bytes 2024-09-15T00:51:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 188022784 bytes 2024-09-15T00:51:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 326897664 bytes 2024-09-15T00:51:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 362264576 bytes 2024-09-15T00:51:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 420009984 bytes 2024-09-15T00:51:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 486853632 bytes 2024-09-15T00:51:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 518403072 bytes 2024-09-15T00:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 551234560 bytes 2024-09-15T00:51:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 649742336 bytes 2024-09-15T00:51:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 753754112 bytes 2024-09-15T00:51:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 798554112 bytes 2024-09-15T00:51:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 856357888 bytes 2024-09-15T00:51:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 870396928 bytes 2024-09-15T00:51:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 876764160 bytes 2024-09-15T00:51:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 932901888 bytes 2024-09-15T00:51:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 959174656 bytes 2024-09-15T00:51:45.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1007118336 bytes 2024-09-15T00:51:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1040742400 bytes 2024-09-15T00:51:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1070867456 bytes 2024-09-15T00:51:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1124367360 bytes 2024-09-15T00:51:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1149285376 bytes 2024-09-15T00:51:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1203246080 bytes 2024-09-15T00:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1217734656 bytes 2024-09-15T00:51:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1257756672 bytes 2024-09-15T00:51:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1283166208 bytes 2024-09-15T00:51:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1326908416 bytes 2024-09-15T00:51:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1394289664 bytes 2024-09-15T00:51:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1429752832 bytes 2024-09-15T00:51:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1487427584 bytes 2024-09-15T00:51:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1506144256 bytes 2024-09-15T00:51:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1516733440 bytes 2024-09-15T00:51:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1565128704 bytes 2024-09-15T00:51:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1651488768 bytes 2024-09-15T00:51:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1731393536 bytes 2024-09-15T00:51:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1764051968 bytes 2024-09-15T00:51:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1796508672 bytes 2024-09-15T00:51:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1818691584 bytes 2024-09-15T00:51:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1868198912 bytes 2024-09-15T00:51:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1879514112 bytes 2024-09-15T00:51:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1929230336 bytes 2024-09-15T00:51:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2011188224 bytes 2024-09-15T00:51:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2022775808 bytes 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_31 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:51:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:51:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:51:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2024-09-15T00:51:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:51:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:51:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-15T00:51:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:51:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-15T00:51:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:51:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-15T00:51:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_31': 2024-09-15T00:51:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:51:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:51:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:51:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13e9e5280da2 2024-09-15T00:51:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13e9e5280da2 2024-09-15T00:51:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:51:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:51:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:51:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:51:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:48 2024 2024-09-15T00:51:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:48 2024 2024-09-15T00:51:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:48 2024 2024-09-15T00:51:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:51:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:51:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 817ee7fa-f621-4e9d-a55c-56e846b6b7f5 2024-09-15T00:51:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:51:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:51:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-15T00:51:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:51:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:51:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=9 2024-09-15T00:51:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:51:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:51:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 65274880 bytes 2024-09-15T00:51:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 90977280 bytes 2024-09-15T00:51:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 141270016 bytes 2024-09-15T00:51:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 173749248 bytes 2024-09-15T00:51:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 243914752 bytes 2024-09-15T00:51:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 311948288 bytes 2024-09-15T00:51:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 357182464 bytes 2024-09-15T00:51:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 452293632 bytes 2024-09-15T00:51:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 491075584 bytes 2024-09-15T00:51:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 572368896 bytes 2024-09-15T00:51:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 632535040 bytes 2024-09-15T00:51:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 633157632 bytes 2024-09-15T00:51:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 678451200 bytes 2024-09-15T00:51:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 679060480 bytes 2024-09-15T00:51:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 708079616 bytes 2024-09-15T00:51:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 745492480 bytes 2024-09-15T00:51:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 817487872 bytes 2024-09-15T00:51:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 855947264 bytes 2024-09-15T00:51:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 910398464 bytes 2024-09-15T00:51:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 941733888 bytes 2024-09-15T00:51:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 983352320 bytes 2024-09-15T00:51:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1029451776 bytes 2024-09-15T00:51:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1030075392 bytes 2024-09-15T00:51:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1062404096 bytes 2024-09-15T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1089647616 bytes 2024-09-15T00:51:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1105972224 bytes 2024-09-15T00:51:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1134460928 bytes 2024-09-15T00:51:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1194851328 bytes 2024-09-15T00:51:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1245797376 bytes 2024-09-15T00:51:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1278104576 bytes 2024-09-15T00:51:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1334060032 bytes 2024-09-15T00:51:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1334680576 bytes 2024-09-15T00:51:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1356046336 bytes 2024-09-15T00:51:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1361438720 bytes 2024-09-15T00:51:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1436293120 bytes 2024-09-15T00:51:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1500974080 bytes 2024-09-15T00:51:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1537224704 bytes 2024-09-15T00:51:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1574218752 bytes 2024-09-15T00:51:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1609046016 bytes 2024-09-15T00:51:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1660786688 bytes 2024-09-15T00:51:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1730885632 bytes 2024-09-15T00:51:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1731502080 bytes 2024-09-15T00:51:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1774938112 bytes 2024-09-15T00:51:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1808253952 bytes 2024-09-15T00:51:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1843812352 bytes 2024-09-15T00:51:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1844435968 bytes 2024-09-15T00:51:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1861207040 bytes 2024-09-15T00:51:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1895571456 bytes 2024-09-15T00:51:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1969347584 bytes 2024-09-15T00:51:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2016768000 bytes 2024-09-15T00:51:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2031383552 bytes 2024-09-15T00:51:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2033112064 bytes 2024-09-15T00:51:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2062991360 bytes 2024-09-15T00:51:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2063632384 bytes 2024-09-15T00:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2180726784 bytes 2024-09-15T00:51:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2251823104 bytes 2024-09-15T00:51:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2354613248 bytes 2024-09-15T00:51:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2429135872 bytes 2024-09-15T00:51:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2458264576 bytes 2024-09-15T00:51:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2490510336 bytes 2024-09-15T00:51:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2535730176 bytes 2024-09-15T00:51:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2601876480 bytes 2024-09-15T00:51:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2623842304 bytes 2024-09-15T00:51:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2673356800 bytes 2024-09-15T00:51:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2717389824 bytes 2024-09-15T00:51:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2746122240 bytes 2024-09-15T00:51:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2786676736 bytes 2024-09-15T00:51:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2787295232 bytes 2024-09-15T00:51:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2787894272 bytes 2024-09-15T00:51:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2840475648 bytes 2024-09-15T00:51:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2854295552 bytes 2024-09-15T00:51:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2867971072 bytes 2024-09-15T00:51:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:51:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:51:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:51:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:51:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_32 2024-09-15T00:51:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-15T00:51:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:51:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:51:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:51:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:51:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:51:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2024-09-15T00:51:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:51:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:51:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:51:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-15T00:51:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-15T00:51:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:51:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-15T00:51:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_32': 2024-09-15T00:51:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:51:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:51:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:51:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 13f5e7f30480 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.13f5e7f30480 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:58 2024 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:58 2024 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:58 2024 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: cf355f92-98e8-41b9-8a17-84b24e81d482 2024-09-15T00:51:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:51:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:51:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-15T00:51:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:51:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:51:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:51:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:51:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=12 2024-09-15T00:51:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:51:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:51:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 13601792 bytes 2024-09-15T00:51:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 55843840 bytes 2024-09-15T00:51:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 137325568 bytes 2024-09-15T00:51:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 169378816 bytes 2024-09-15T00:51:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 169969664 bytes 2024-09-15T00:51:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 191647744 bytes 2024-09-15T00:51:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 281711616 bytes 2024-09-15T00:51:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 321942528 bytes 2024-09-15T00:51:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 380029952 bytes 2024-09-15T00:52:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 444833792 bytes 2024-09-15T00:52:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 500988928 bytes 2024-09-15T00:52:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 516702208 bytes 2024-09-15T00:52:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 539465728 bytes 2024-09-15T00:52:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 640948224 bytes 2024-09-15T00:52:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 663880704 bytes 2024-09-15T00:52:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 691320832 bytes 2024-09-15T00:52:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 742612992 bytes 2024-09-15T00:52:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 752200704 bytes 2024-09-15T00:52:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 787894272 bytes 2024-09-15T00:52:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 840649728 bytes 2024-09-15T00:52:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 889524224 bytes 2024-09-15T00:52:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 919302144 bytes 2024-09-15T00:52:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 966291456 bytes 2024-09-15T00:52:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1001042944 bytes 2024-09-15T00:52:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1019282432 bytes 2024-09-15T00:52:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1036453888 bytes 2024-09-15T00:52:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1111225344 bytes 2024-09-15T00:52:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1175123968 bytes 2024-09-15T00:52:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1225927680 bytes 2024-09-15T00:52:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1236899840 bytes 2024-09-15T00:52:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1252218880 bytes 2024-09-15T00:52:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1290731520 bytes 2024-09-15T00:52:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1348683776 bytes 2024-09-15T00:52:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1420048384 bytes 2024-09-15T00:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1456331776 bytes 2024-09-15T00:52:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1574802432 bytes 2024-09-15T00:52:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1623790592 bytes 2024-09-15T00:52:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1663265792 bytes 2024-09-15T00:52:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1688698880 bytes 2024-09-15T00:52:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1705097216 bytes 2024-09-15T00:52:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1721284608 bytes 2024-09-15T00:52:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1768881152 bytes 2024-09-15T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1860235264 bytes 2024-09-15T00:52:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1894641664 bytes 2024-09-15T00:52:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1944619008 bytes 2024-09-15T00:52:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1966307328 bytes 2024-09-15T00:52:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2014623744 bytes 2024-09-15T00:52:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2085165056 bytes 2024-09-15T00:52:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2118525952 bytes 2024-09-15T00:52:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2119143424 bytes 2024-09-15T00:52:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2153136128 bytes 2024-09-15T00:52:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2204356608 bytes 2024-09-15T00:52:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2264015872 bytes 2024-09-15T00:52:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2305975296 bytes 2024-09-15T00:52:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2354861056 bytes 2024-09-15T00:52:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2409731072 bytes 2024-09-15T00:52:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2472869888 bytes 2024-09-15T00:52:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2473460736 bytes 2024-09-15T00:52:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2512653312 bytes 2024-09-15T00:52:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2545899520 bytes 2024-09-15T00:52:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2598294528 bytes 2024-09-15T00:52:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2623569920 bytes 2024-09-15T00:52:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2645390336 bytes 2024-09-15T00:52:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2768620544 bytes 2024-09-15T00:52:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2879611904 bytes 2024-09-15T00:52:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2901284864 bytes 2024-09-15T00:52:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2949650432 bytes 2024-09-15T00:52:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2956857344 bytes 2024-09-15T00:52:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2976048128 bytes 2024-09-15T00:52:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3037230080 bytes 2024-09-15T00:52:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3073862656 bytes 2024-09-15T00:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3090694144 bytes 2024-09-15T00:52:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3120676864 bytes 2024-09-15T00:52:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3136112640 bytes 2024-09-15T00:52:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3160778752 bytes 2024-09-15T00:52:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3179249664 bytes 2024-09-15T00:52:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3184954368 bytes 2024-09-15T00:52:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3189372928 bytes 2024-09-15T00:52:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3246807040 bytes 2024-09-15T00:52:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3251731456 bytes 2024-09-15T00:52:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3283448832 bytes 2024-09-15T00:52:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3361828864 bytes 2024-09-15T00:52:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3434613760 bytes 2024-09-15T00:52:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3468688384 bytes 2024-09-15T00:52:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3492855808 bytes 2024-09-15T00:52:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3495388160 bytes 2024-09-15T00:52:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3522348032 bytes 2024-09-15T00:52:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3522979840 bytes 2024-09-15T00:52:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3553841152 bytes 2024-09-15T00:52:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3603240960 bytes 2024-09-15T00:52:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3664723968 bytes 2024-09-15T00:52:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3672431616 bytes 2024-09-15T00:52:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3693472768 bytes 2024-09-15T00:52:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3742320640 bytes 2024-09-15T00:52:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 3742946304 bytes 2024-09-15T00:52:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 3753160704 bytes 2024-09-15T00:52:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 3800140800 bytes 2024-09-15T00:52:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:52:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:52:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_33 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-15T00:52:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:52:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:52:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:52:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:52:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2024-09-15T00:52:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:52:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:52:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:52:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-15T00:52:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:52:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:52:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-15T00:52:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:52:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-15T00:52:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_33': 2024-09-15T00:52:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:52:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:52:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1401e95486f1 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1401e95486f1 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:10 2024 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:10 2024 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:10 2024 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:52:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 81151a5e-5338-4e8e-8ee9-a1dc81c31cc6 2024-09-15T00:52:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:52:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:52:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-15T00:52:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:52:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:52:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=6 2024-09-15T00:52:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:52:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:52:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 38168576 bytes 2024-09-15T00:52:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 66965504 bytes 2024-09-15T00:52:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 103980032 bytes 2024-09-15T00:52:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 158782464 bytes 2024-09-15T00:52:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 213791744 bytes 2024-09-15T00:52:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 232956928 bytes 2024-09-15T00:52:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 233565184 bytes 2024-09-15T00:52:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 282265600 bytes 2024-09-15T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 389578752 bytes 2024-09-15T00:52:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 452460544 bytes 2024-09-15T00:52:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 512047104 bytes 2024-09-15T00:52:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 547172352 bytes 2024-09-15T00:52:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 564578304 bytes 2024-09-15T00:52:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 603400192 bytes 2024-09-15T00:52:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 664964096 bytes 2024-09-15T00:52:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 768333824 bytes 2024-09-15T00:52:14.045 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7649953890316353) > weight (0.25) 2024-09-15T00:52:14.046 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.524076903518837) > weight (0.25) 2024-09-15T00:52:14.046 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.654923995622351) > weight (0.25) 2024-09-15T00:52:14.046 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.388807243302728) > weight (0.25) 2024-09-15T00:52:14.046 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-15T00:52:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 768925696 bytes 2024-09-15T00:52:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 810706944 bytes 2024-09-15T00:52:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 860948480 bytes 2024-09-15T00:52:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 904196096 bytes 2024-09-15T00:52:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 941871104 bytes 2024-09-15T00:52:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 985259008 bytes 2024-09-15T00:52:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1039616000 bytes 2024-09-15T00:52:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1079049216 bytes 2024-09-15T00:52:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1106810880 bytes 2024-09-15T00:52:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1135650816 bytes 2024-09-15T00:52:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1205054464 bytes 2024-09-15T00:52:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1275499520 bytes 2024-09-15T00:52:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1288798208 bytes 2024-09-15T00:52:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1362031616 bytes 2024-09-15T00:52:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1465584640 bytes 2024-09-15T00:52:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1496030208 bytes 2024-09-15T00:52:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1513089024 bytes 2024-09-15T00:52:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1544413184 bytes 2024-09-15T00:52:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1596020736 bytes 2024-09-15T00:52:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1678334976 bytes 2024-09-15T00:52:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1682018304 bytes 2024-09-15T00:52:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1752392704 bytes 2024-09-15T00:52:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1788793856 bytes 2024-09-15T00:52:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1839270912 bytes 2024-09-15T00:52:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1842462720 bytes 2024-09-15T00:52:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1885318144 bytes 2024-09-15T00:52:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1917333504 bytes 2024-09-15T00:52:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1975829504 bytes 2024-09-15T00:52:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2065763328 bytes 2024-09-15T00:52:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2100520960 bytes 2024-09-15T00:52:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:52:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:52:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:52:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:52:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:52:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_34 2024-09-15T00:52:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-15T00:52:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:52:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:52:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:52:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2024-09-15T00:52:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:52:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:52:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:52:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-15T00:52:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:52:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:52:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-15T00:52:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:52:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-15T00:52:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_34': 2024-09-15T00:52:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:52:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:52:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:52:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 140d9db4bbd9 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.140d9db4bbd9 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:17 2024 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:17 2024 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:17 2024 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: b1e366ef-a777-4914-8fb6-5426e0f998dd 2024-09-15T00:52:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:52:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:52:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-15T00:52:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:52:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:52:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=9 2024-09-15T00:52:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:52:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:52:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 75112448 bytes 2024-09-15T00:52:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 134867968 bytes 2024-09-15T00:52:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 140963840 bytes 2024-09-15T00:52:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 152225792 bytes 2024-09-15T00:52:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 198647808 bytes 2024-09-15T00:52:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 210566144 bytes 2024-09-15T00:52:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 249141248 bytes 2024-09-15T00:52:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 274231296 bytes 2024-09-15T00:52:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 333805568 bytes 2024-09-15T00:52:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 391421952 bytes 2024-09-15T00:52:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 428211200 bytes 2024-09-15T00:52:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 479222784 bytes 2024-09-15T00:52:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 480909312 bytes 2024-09-15T00:52:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 491086848 bytes 2024-09-15T00:52:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 529216512 bytes 2024-09-15T00:52:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 568315904 bytes 2024-09-15T00:52:20.831 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-15T00:52:20.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-15T00:52:20.831 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-15T00:52:20.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T00:52:20.831+0000 7ff101ca2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 45976) UID: 1000 2024-09-15T00:52:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 582137856 bytes 2024-09-15T00:52:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 587330560 bytes 2024-09-15T00:52:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 604363776 bytes 2024-09-15T00:52:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 675275776 bytes 2024-09-15T00:52:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 743393280 bytes 2024-09-15T00:52:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 786269184 bytes 2024-09-15T00:52:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 809525248 bytes 2024-09-15T00:52:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 857392128 bytes 2024-09-15T00:52:21.832 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-15T00:52:21.832 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:52:21.832 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-15T00:52:21.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-15T00:52:21.833 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-15T00:52:21.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-15T00:52:21.836 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-09-15T00:52:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 908835840 bytes 2024-09-15T00:52:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 924546048 bytes 2024-09-15T00:52:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 947621888 bytes 2024-09-15T00:52:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1003747328 bytes 2024-09-15T00:52:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1027823616 bytes 2024-09-15T00:52:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1032180736 bytes 2024-09-15T00:52:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1102794752 bytes 2024-09-15T00:52:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1157367808 bytes 2024-09-15T00:52:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1199789056 bytes 2024-09-15T00:52:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1215142912 bytes 2024-09-15T00:52:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1301061632 bytes 2024-09-15T00:52:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1301655552 bytes 2024-09-15T00:52:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1326648320 bytes 2024-09-15T00:52:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1367806976 bytes 2024-09-15T00:52:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1400420352 bytes 2024-09-15T00:52:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1440338944 bytes 2024-09-15T00:52:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1458695168 bytes 2024-09-15T00:52:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1533996032 bytes 2024-09-15T00:52:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1553449984 bytes 2024-09-15T00:52:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1627752448 bytes 2024-09-15T00:52:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1673732096 bytes 2024-09-15T00:52:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1729603584 bytes 2024-09-15T00:52:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1752862720 bytes 2024-09-15T00:52:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1797765120 bytes 2024-09-15T00:52:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1852797952 bytes 2024-09-15T00:52:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1901750272 bytes 2024-09-15T00:52:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1926225920 bytes 2024-09-15T00:52:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 1960379392 bytes 2024-09-15T00:52:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2007344128 bytes 2024-09-15T00:52:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2028141568 bytes 2024-09-15T00:52:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2063885312 bytes 2024-09-15T00:52:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2090442752 bytes 2024-09-15T00:52:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2159692800 bytes 2024-09-15T00:52:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2238470144 bytes 2024-09-15T00:52:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2259828736 bytes 2024-09-15T00:52:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2291269632 bytes 2024-09-15T00:52:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2344258560 bytes 2024-09-15T00:52:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2380237824 bytes 2024-09-15T00:52:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2446394368 bytes 2024-09-15T00:52:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2530976768 bytes 2024-09-15T00:52:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2571017216 bytes 2024-09-15T00:52:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2588208128 bytes 2024-09-15T00:52:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2600556544 bytes 2024-09-15T00:52:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2658609152 bytes 2024-09-15T00:52:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2701486080 bytes 2024-09-15T00:52:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2757554176 bytes 2024-09-15T00:52:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2799769600 bytes 2024-09-15T00:52:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2828916736 bytes 2024-09-15T00:52:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:52:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:52:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:52:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:52:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_35 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-15T00:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T00:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2024-09-15T00:52:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:52:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:52:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:52:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-15T00:52:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T00:52:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:52:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-15T00:52:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:52:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-15T00:52:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_35': 2024-09-15T00:52:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:52:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:52:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:52:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 141fdab45bfb 2024-09-15T00:52:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.141fdab45bfb 2024-09-15T00:52:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:52:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:52:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:52:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:52:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:27 2024 2024-09-15T00:52:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:27 2024 2024-09-15T00:52:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:27 2024 2024-09-15T00:52:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:52:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:52:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 264c4375-3155-4e86-9175-01a1a2ca5c03 2024-09-15T00:52:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:52:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:52:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-15T00:52:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T00:52:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:52:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=12 2024-09-15T00:52:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:52:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:52:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 92826624 bytes 2024-09-15T00:52:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 115645440 bytes 2024-09-15T00:52:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 133792768 bytes 2024-09-15T00:52:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 141394944 bytes 2024-09-15T00:52:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 238189568 bytes 2024-09-15T00:52:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 292767744 bytes 2024-09-15T00:52:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 293374976 bytes 2024-09-15T00:52:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 361201664 bytes 2024-09-15T00:52:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 442848256 bytes 2024-09-15T00:52:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 524540928 bytes 2024-09-15T00:52:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 576294912 bytes 2024-09-15T00:52:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 605003776 bytes 2024-09-15T00:52:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 692096000 bytes 2024-09-15T00:52:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 746347520 bytes 2024-09-15T00:52:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 777179136 bytes 2024-09-15T00:52:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 807156736 bytes 2024-09-15T00:52:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 873226240 bytes 2024-09-15T00:52:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 900531200 bytes 2024-09-15T00:52:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 1007969280 bytes 2024-09-15T00:52:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1083833344 bytes 2024-09-15T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1111743488 bytes 2024-09-15T00:52:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1122597888 bytes 2024-09-15T00:52:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1133507584 bytes 2024-09-15T00:52:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1149982720 bytes 2024-09-15T00:52:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1206519808 bytes 2024-09-15T00:52:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1281165312 bytes 2024-09-15T00:52:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1281792000 bytes 2024-09-15T00:52:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1338890240 bytes 2024-09-15T00:52:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1420436480 bytes 2024-09-15T00:52:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1469361152 bytes 2024-09-15T00:52:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1526964224 bytes 2024-09-15T00:52:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1549179904 bytes 2024-09-15T00:52:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1557330944 bytes 2024-09-15T00:52:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1594152960 bytes 2024-09-15T00:52:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1597851648 bytes 2024-09-15T00:52:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1658935296 bytes 2024-09-15T00:52:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1768901632 bytes 2024-09-15T00:52:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1811181568 bytes 2024-09-15T00:52:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1816313856 bytes 2024-09-15T00:52:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1840199680 bytes 2024-09-15T00:52:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1927074816 bytes 2024-09-15T00:52:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1961247744 bytes 2024-09-15T00:52:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1997012992 bytes 2024-09-15T00:52:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2061466624 bytes 2024-09-15T00:52:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2097427456 bytes 2024-09-15T00:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2179554304 bytes 2024-09-15T00:52:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2195007488 bytes 2024-09-15T00:52:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2203848704 bytes 2024-09-15T00:52:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2220770304 bytes 2024-09-15T00:52:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2306029568 bytes 2024-09-15T00:52:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2314792960 bytes 2024-09-15T00:52:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2329778176 bytes 2024-09-15T00:52:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2389453824 bytes 2024-09-15T00:52:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2423420928 bytes 2024-09-15T00:52:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2447221760 bytes 2024-09-15T00:52:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2540369920 bytes 2024-09-15T00:52:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2590535680 bytes 2024-09-15T00:52:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2649508864 bytes 2024-09-15T00:52:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2761373696 bytes 2024-09-15T00:52:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2780751872 bytes 2024-09-15T00:52:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2794150912 bytes 2024-09-15T00:52:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2891561984 bytes 2024-09-15T00:52:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2958507008 bytes 2024-09-15T00:52:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2971112448 bytes 2024-09-15T00:52:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2989351936 bytes 2024-09-15T00:52:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2992787456 bytes 2024-09-15T00:52:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3006885888 bytes 2024-09-15T00:52:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3098373120 bytes 2024-09-15T00:52:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3178447872 bytes 2024-09-15T00:52:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3244862464 bytes 2024-09-15T00:52:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3380183040 bytes 2024-09-15T00:52:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3401830400 bytes 2024-09-15T00:52:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3418384384 bytes 2024-09-15T00:52:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3436767232 bytes 2024-09-15T00:52:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3495901184 bytes 2024-09-15T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3574185984 bytes 2024-09-15T00:52:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3617354752 bytes 2024-09-15T00:52:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3630732288 bytes 2024-09-15T00:52:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3698652160 bytes 2024-09-15T00:52:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3699277824 bytes 2024-09-15T00:52:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3742312448 bytes 2024-09-15T00:52:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3848261632 bytes 2024-09-15T00:52:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3891214336 bytes 2024-09-15T00:52:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3934286848 bytes 2024-09-15T00:52:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3959647232 bytes 2024-09-15T00:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 4020187136 bytes 2024-09-15T00:52:40.098 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:52:40.096+0000 7f9966144640 -1 reset not still connected to 0x55ef586749c0 2024-09-15T00:52:40.098 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:52:40.096+0000 7f9966144640 -1 reset not still connected to 0x55ef5af25ee0 2024-09-15T00:52:40.098 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:52:40.096+0000 7f9966144640 -1 reset not still connected to 0x55ef5c2d7a00 2024-09-15T00:52:40.098 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:52:40.096+0000 7f9966144640 -1 reset not still connected to 0x55ef5d565450 2024-09-15T00:52:40.099 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:52:40.097+0000 7f9966144640 -1 reset not still connected to 0x55ef5ecde9c0 2024-09-15T00:52:40.099 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:52:40.097+0000 7f9966144640 -1 reset not still connected to 0x55ef6186d6c0 2024-09-15T00:52:40.099 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:52:40.097+0000 7f9966144640 -1 reset not still connected to 0x55ef675e8410 2024-09-15T00:52:40.099 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:52:40.097+0000 7f9966144640 -1 reset not still connected to 0x55ef6facd930 2024-09-15T00:52:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 4027087872 bytes 2024-09-15T00:52:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 4096942080 bytes 2024-09-15T00:52:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 4166959104 bytes 2024-09-15T00:52:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 4191975424 bytes 2024-09-15T00:52:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 4258631680 bytes 2024-09-15T00:52:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 4325135360 bytes 2024-09-15T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_36 2024-09-15T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-15T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:52:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-15T00:52:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T00:52:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:52:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:52:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:52:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2024-09-15T00:52:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:52:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:52:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:52:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-15T00:52:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T00:52:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:52:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-15T00:52:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:52:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_36': 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 142b8b058c74 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.142b8b058c74 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:52:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:40 2024 2024-09-15T00:52:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:40 2024 2024-09-15T00:52:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:40 2024 2024-09-15T00:52:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:52:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:52:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: c97b3f87-8bd7-4173-9bd5-d9d0df48b80a 2024-09-15T00:52:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:52:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:52:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-15T00:52:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T00:52:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:52:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=13 2024-09-15T00:52:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:52:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:52:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 58935296 bytes 2024-09-15T00:52:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 100774912 bytes 2024-09-15T00:52:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 148117504 bytes 2024-09-15T00:52:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 182702080 bytes 2024-09-15T00:52:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 276504576 bytes 2024-09-15T00:52:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 284214272 bytes 2024-09-15T00:52:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 288612352 bytes 2024-09-15T00:52:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 348032000 bytes 2024-09-15T00:52:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 417110016 bytes 2024-09-15T00:52:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 475485184 bytes 2024-09-15T00:52:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 487183360 bytes 2024-09-15T00:52:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 532402176 bytes 2024-09-15T00:52:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 556414976 bytes 2024-09-15T00:52:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 599410688 bytes 2024-09-15T00:52:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 648039424 bytes 2024-09-15T00:52:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 686170112 bytes 2024-09-15T00:52:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 704097280 bytes 2024-09-15T00:52:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 756154368 bytes 2024-09-15T00:52:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 837170176 bytes 2024-09-15T00:52:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 952797184 bytes 2024-09-15T00:52:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 968387584 bytes 2024-09-15T00:52:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 969011200 bytes 2024-09-15T00:52:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1002657792 bytes 2024-09-15T00:52:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1074724864 bytes 2024-09-15T00:52:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1103982592 bytes 2024-09-15T00:52:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1116882944 bytes 2024-09-15T00:52:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1172809728 bytes 2024-09-15T00:52:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1178988544 bytes 2024-09-15T00:52:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1181158400 bytes 2024-09-15T00:52:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1255331840 bytes 2024-09-15T00:52:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1255919616 bytes 2024-09-15T00:52:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1256527872 bytes 2024-09-15T00:52:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1269102592 bytes 2024-09-15T00:52:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1305158656 bytes 2024-09-15T00:52:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1324504064 bytes 2024-09-15T00:52:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1383923712 bytes 2024-09-15T00:52:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1459921920 bytes 2024-09-15T00:52:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1465852928 bytes 2024-09-15T00:52:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1498403840 bytes 2024-09-15T00:52:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1547601920 bytes 2024-09-15T00:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1578637312 bytes 2024-09-15T00:52:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1647142912 bytes 2024-09-15T00:52:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1698673664 bytes 2024-09-15T00:52:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1737011200 bytes 2024-09-15T00:52:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1737631744 bytes 2024-09-15T00:52:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1759705088 bytes 2024-09-15T00:52:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1796166656 bytes 2024-09-15T00:52:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1850024960 bytes 2024-09-15T00:52:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1890877440 bytes 2024-09-15T00:52:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1895200768 bytes 2024-09-15T00:52:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1932441600 bytes 2024-09-15T00:52:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 1971438592 bytes 2024-09-15T00:52:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2035704832 bytes 2024-09-15T00:52:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2106379264 bytes 2024-09-15T00:52:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2152780800 bytes 2024-09-15T00:52:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2179106816 bytes 2024-09-15T00:52:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2218680320 bytes 2024-09-15T00:52:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2226390016 bytes 2024-09-15T00:52:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2255770624 bytes 2024-09-15T00:52:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2268780544 bytes 2024-09-15T00:52:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2284974080 bytes 2024-09-15T00:52:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2327498752 bytes 2024-09-15T00:52:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2378812416 bytes 2024-09-15T00:52:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2431954944 bytes 2024-09-15T00:52:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2460477440 bytes 2024-09-15T00:52:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2476724224 bytes 2024-09-15T00:52:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2537162752 bytes 2024-09-15T00:52:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2552485888 bytes 2024-09-15T00:52:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2560654336 bytes 2024-09-15T00:52:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2646846464 bytes 2024-09-15T00:52:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2702126080 bytes 2024-09-15T00:52:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2730213376 bytes 2024-09-15T00:52:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2757633024 bytes 2024-09-15T00:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2778051584 bytes 2024-09-15T00:52:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2808693760 bytes 2024-09-15T00:52:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2838663168 bytes 2024-09-15T00:52:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 2882754560 bytes 2024-09-15T00:52:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 2957370368 bytes 2024-09-15T00:52:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 2984857600 bytes 2024-09-15T00:52:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 2985466880 bytes 2024-09-15T00:52:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3017366528 bytes 2024-09-15T00:52:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3078318080 bytes 2024-09-15T00:52:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3101409280 bytes 2024-09-15T00:52:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3185359872 bytes 2024-09-15T00:52:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3206802432 bytes 2024-09-15T00:52:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3234126848 bytes 2024-09-15T00:52:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3266626560 bytes 2024-09-15T00:52:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3298326528 bytes 2024-09-15T00:52:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3346432000 bytes 2024-09-15T00:52:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3403905024 bytes 2024-09-15T00:52:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3468275712 bytes 2024-09-15T00:52:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3541513216 bytes 2024-09-15T00:52:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3557444608 bytes 2024-09-15T00:52:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3623890944 bytes 2024-09-15T00:52:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 3653805056 bytes 2024-09-15T00:52:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 3696731136 bytes 2024-09-15T00:52:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 3759254528 bytes 2024-09-15T00:52:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 3783079936 bytes 2024-09-15T00:52:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 3842608128 bytes 2024-09-15T00:52:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 3860644864 bytes 2024-09-15T00:52:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 3935100928 bytes 2024-09-15T00:52:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 3977115648 bytes 2024-09-15T00:52:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4066086912 bytes 2024-09-15T00:52:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4105675776 bytes 2024-09-15T00:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4141358080 bytes 2024-09-15T00:52:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:52:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:52:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:52:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:52:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:52:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_37 2024-09-15T00:52:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-15T00:52:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T00:52:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:52:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:52:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:52:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:52:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2024-09-15T00:52:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:52:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:52:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:52:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-15T00:52:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T00:52:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:52:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-15T00:52:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:52:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-15T00:52:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_37': 2024-09-15T00:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 14376ea7c2e7 2024-09-15T00:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.14376ea7c2e7 2024-09-15T00:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:52:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:55 2024 2024-09-15T00:52:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:55 2024 2024-09-15T00:52:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:55 2024 2024-09-15T00:52:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:52:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:52:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 45040e2c-2dd5-4e27-8546-d207a9f5ca1e 2024-09-15T00:52:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:52:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:52:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-15T00:52:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:52:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:52:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T00:52:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=14 2024-09-15T00:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:52:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 30951424 bytes 2024-09-15T00:52:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 61218816 bytes 2024-09-15T00:52:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 103232512 bytes 2024-09-15T00:52:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 113426432 bytes 2024-09-15T00:52:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 147771392 bytes 2024-09-15T00:52:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 148377600 bytes 2024-09-15T00:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 171293696 bytes 2024-09-15T00:52:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 209681408 bytes 2024-09-15T00:52:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 279631872 bytes 2024-09-15T00:52:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 301624320 bytes 2024-09-15T00:52:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 334355456 bytes 2024-09-15T00:52:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 355017728 bytes 2024-09-15T00:52:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 420024320 bytes 2024-09-15T00:52:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 470165504 bytes 2024-09-15T00:52:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 480689152 bytes 2024-09-15T00:52:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 583750656 bytes 2024-09-15T00:52:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 614572032 bytes 2024-09-15T00:52:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 630382592 bytes 2024-09-15T00:52:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 707718144 bytes 2024-09-15T00:52:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 737257472 bytes 2024-09-15T00:52:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 787971072 bytes 2024-09-15T00:52:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 814766080 bytes 2024-09-15T00:52:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 832359424 bytes 2024-09-15T00:52:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 909780992 bytes 2024-09-15T00:52:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 980459520 bytes 2024-09-15T00:52:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1024777216 bytes 2024-09-15T00:52:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1085258752 bytes 2024-09-15T00:53:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1105524736 bytes 2024-09-15T00:53:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1168499712 bytes 2024-09-15T00:53:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1231057920 bytes 2024-09-15T00:53:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1293599744 bytes 2024-09-15T00:53:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1391302656 bytes 2024-09-15T00:53:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1434890240 bytes 2024-09-15T00:53:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1522928640 bytes 2024-09-15T00:53:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1590354944 bytes 2024-09-15T00:53:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1648932864 bytes 2024-09-15T00:53:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1657358336 bytes 2024-09-15T00:53:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1705380864 bytes 2024-09-15T00:53:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1735829504 bytes 2024-09-15T00:53:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1755516928 bytes 2024-09-15T00:53:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1784778752 bytes 2024-09-15T00:53:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1822000128 bytes 2024-09-15T00:53:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1849262080 bytes 2024-09-15T00:53:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1881812992 bytes 2024-09-15T00:53:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1968743424 bytes 2024-09-15T00:53:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2015095808 bytes 2024-09-15T00:53:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2016390144 bytes 2024-09-15T00:53:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2119137280 bytes 2024-09-15T00:53:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2164946944 bytes 2024-09-15T00:53:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2203417600 bytes 2024-09-15T00:53:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2222867456 bytes 2024-09-15T00:53:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2268063744 bytes 2024-09-15T00:53:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2355014656 bytes 2024-09-15T00:53:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2402197504 bytes 2024-09-15T00:53:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2440163328 bytes 2024-09-15T00:53:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2440800256 bytes 2024-09-15T00:53:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2457801728 bytes 2024-09-15T00:53:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2481061888 bytes 2024-09-15T00:53:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2512007168 bytes 2024-09-15T00:53:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2578944000 bytes 2024-09-15T00:53:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2614727680 bytes 2024-09-15T00:53:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2648139776 bytes 2024-09-15T00:53:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2715052032 bytes 2024-09-15T00:53:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2750100480 bytes 2024-09-15T00:53:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2840999936 bytes 2024-09-15T00:53:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2898793472 bytes 2024-09-15T00:53:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2963863552 bytes 2024-09-15T00:53:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3033990144 bytes 2024-09-15T00:53:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3064469504 bytes 2024-09-15T00:53:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3099312128 bytes 2024-09-15T00:53:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3141947392 bytes 2024-09-15T00:53:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3204356096 bytes 2024-09-15T00:53:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3242745856 bytes 2024-09-15T00:53:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3331211264 bytes 2024-09-15T00:53:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3368957952 bytes 2024-09-15T00:53:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3456189440 bytes 2024-09-15T00:53:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3495999488 bytes 2024-09-15T00:53:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3560333312 bytes 2024-09-15T00:53:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3640162304 bytes 2024-09-15T00:53:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3713998848 bytes 2024-09-15T00:53:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3728547840 bytes 2024-09-15T00:53:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3743194112 bytes 2024-09-15T00:53:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3798717440 bytes 2024-09-15T00:53:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3819034624 bytes 2024-09-15T00:53:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3943447552 bytes 2024-09-15T00:53:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 4024726528 bytes 2024-09-15T00:53:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 4057106432 bytes 2024-09-15T00:53:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 4130855936 bytes 2024-09-15T00:53:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 4167346176 bytes 2024-09-15T00:53:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 4225468416 bytes 2024-09-15T00:53:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 4287764480 bytes 2024-09-15T00:53:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 4350031872 bytes 2024-09-15T00:53:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 4368588800 bytes 2024-09-15T00:53:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4443873280 bytes 2024-09-15T00:53:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4483033088 bytes 2024-09-15T00:53:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4545558528 bytes 2024-09-15T00:53:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4598544384 bytes 2024-09-15T00:53:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4638209024 bytes 2024-09-15T00:53:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4701942784 bytes 2024-09-15T00:53:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4723824640 bytes 2024-09-15T00:53:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4737469440 bytes 2024-09-15T00:53:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4773289984 bytes 2024-09-15T00:53:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4825100288 bytes 2024-09-15T00:53:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4873859072 bytes 2024-09-15T00:53:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4906967040 bytes 2024-09-15T00:53:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:53:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:53:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:53:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:53:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_38 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-15T00:53:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T00:53:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:53:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:53:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:53:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2024-09-15T00:53:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:53:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:53:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:53:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-15T00:53:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T00:53:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:53:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-15T00:53:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:53:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-15T00:53:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_38': 2024-09-15T00:53:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1443570a010b 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1443570a010b 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:53:10 2024 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:53:10 2024 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:53:10 2024 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:53:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:53:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: d16734fd-d8ee-40b2-b689-bcc1dad3c46d 2024-09-15T00:53:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:53:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:53:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-15T00:53:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T00:53:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:53:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=11 2024-09-15T00:53:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:53:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:53:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 62534656 bytes 2024-09-15T00:53:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 107695104 bytes 2024-09-15T00:53:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 114499584 bytes 2024-09-15T00:53:12.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 192942080 bytes 2024-09-15T00:53:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 296981504 bytes 2024-09-15T00:53:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 329614336 bytes 2024-09-15T00:53:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 416420864 bytes 2024-09-15T00:53:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 427066368 bytes 2024-09-15T00:53:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 494593024 bytes 2024-09-15T00:53:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 514974720 bytes 2024-09-15T00:53:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 537198592 bytes 2024-09-15T00:53:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 581407744 bytes 2024-09-15T00:53:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 640856064 bytes 2024-09-15T00:53:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 659423232 bytes 2024-09-15T00:53:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 740549632 bytes 2024-09-15T00:53:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 858528768 bytes 2024-09-15T00:53:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 867442688 bytes 2024-09-15T00:53:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 904388608 bytes 2024-09-15T00:53:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 952642560 bytes 2024-09-15T00:53:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1009886208 bytes 2024-09-15T00:53:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1083444224 bytes 2024-09-15T00:53:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1104888832 bytes 2024-09-15T00:53:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1125852160 bytes 2024-09-15T00:53:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1166574592 bytes 2024-09-15T00:53:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1225942016 bytes 2024-09-15T00:53:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1248894976 bytes 2024-09-15T00:53:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1274943488 bytes 2024-09-15T00:53:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1346210816 bytes 2024-09-15T00:53:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1464955904 bytes 2024-09-15T00:53:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1536100352 bytes 2024-09-15T00:53:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1588696064 bytes 2024-09-15T00:53:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1669797888 bytes 2024-09-15T00:53:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1733115904 bytes 2024-09-15T00:53:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1786245120 bytes 2024-09-15T00:53:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1801198592 bytes 2024-09-15T00:53:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1872069632 bytes 2024-09-15T00:53:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1883821056 bytes 2024-09-15T00:53:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1902697472 bytes 2024-09-15T00:53:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1970713600 bytes 2024-09-15T00:53:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 2000269312 bytes 2024-09-15T00:53:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 2034203648 bytes 2024-09-15T00:53:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 2054510592 bytes 2024-09-15T00:53:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2070064128 bytes 2024-09-15T00:53:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2109758464 bytes 2024-09-15T00:53:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2139840512 bytes 2024-09-15T00:53:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2187687936 bytes 2024-09-15T00:53:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2278809600 bytes 2024-09-15T00:53:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2331675648 bytes 2024-09-15T00:53:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2375365632 bytes 2024-09-15T00:53:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2406982656 bytes 2024-09-15T00:53:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2504146944 bytes 2024-09-15T00:53:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2552409088 bytes 2024-09-15T00:53:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2597855232 bytes 2024-09-15T00:53:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2687826944 bytes 2024-09-15T00:53:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2741729280 bytes 2024-09-15T00:53:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2818812928 bytes 2024-09-15T00:53:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2900440064 bytes 2024-09-15T00:53:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2952031232 bytes 2024-09-15T00:53:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2994196480 bytes 2024-09-15T00:53:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 3056900096 bytes 2024-09-15T00:53:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 3129008128 bytes 2024-09-15T00:53:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 3147574272 bytes 2024-09-15T00:53:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 3211402240 bytes 2024-09-15T00:53:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 3241448448 bytes 2024-09-15T00:53:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 3329103872 bytes 2024-09-15T00:53:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 3382409216 bytes 2024-09-15T00:53:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3423616000 bytes 2024-09-15T00:53:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3466220544 bytes 2024-09-15T00:53:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3519598592 bytes 2024-09-15T00:53:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3537258496 bytes 2024-09-15T00:53:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3562512384 bytes 2024-09-15T00:53:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3585424384 bytes 2024-09-15T00:53:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3624679424 bytes 2024-09-15T00:53:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3681897472 bytes 2024-09-15T00:53:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3726197760 bytes 2024-09-15T00:53:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3786507264 bytes 2024-09-15T00:53:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3809810432 bytes 2024-09-15T00:53:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3890108416 bytes 2024-09-15T00:53:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3944713216 bytes 2024-09-15T00:53:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3948408832 bytes 2024-09-15T00:53:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3997619200 bytes 2024-09-15T00:53:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:53:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:53:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:53:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:53:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:53:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_39 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T00:53:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:53:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:53:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:53:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2024-09-15T00:53:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:53:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:53:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:53:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-15T00:53:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T00:53:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:53:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-15T00:53:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:53:23.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-15T00:53:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_39': 2024-09-15T00:53:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:53:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 144fe6f70c8a 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.144fe6f70c8a 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:53:22 2024 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:53:22 2024 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:53:22 2024 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 1dcd151a-a499-468b-b08a-ad4dd40aec05 2024-09-15T00:53:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:53:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:53:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-15T00:53:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T00:53:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:53:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=7 2024-09-15T00:53:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:53:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:53:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 59049984 bytes 2024-09-15T00:53:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 159816704 bytes 2024-09-15T00:53:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 182418432 bytes 2024-09-15T00:53:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 224453632 bytes 2024-09-15T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 298406912 bytes 2024-09-15T00:53:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 343411712 bytes 2024-09-15T00:53:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 368409600 bytes 2024-09-15T00:53:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 393890816 bytes 2024-09-15T00:53:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 448080896 bytes 2024-09-15T00:53:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 475351040 bytes 2024-09-15T00:53:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 498655232 bytes 2024-09-15T00:53:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 506429440 bytes 2024-09-15T00:53:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 534984704 bytes 2024-09-15T00:53:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 540130304 bytes 2024-09-15T00:53:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 540722176 bytes 2024-09-15T00:53:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 582690816 bytes 2024-09-15T00:53:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 626105344 bytes 2024-09-15T00:53:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 639724544 bytes 2024-09-15T00:53:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 710974464 bytes 2024-09-15T00:53:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 729402368 bytes 2024-09-15T00:53:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 754237440 bytes 2024-09-15T00:53:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 875569152 bytes 2024-09-15T00:53:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 891685888 bytes 2024-09-15T00:53:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 973751296 bytes 2024-09-15T00:53:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1038625792 bytes 2024-09-15T00:53:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1103230976 bytes 2024-09-15T00:53:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1142684672 bytes 2024-09-15T00:53:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1186506752 bytes 2024-09-15T00:53:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1206028288 bytes 2024-09-15T00:53:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1233786880 bytes 2024-09-15T00:53:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1293257728 bytes 2024-09-15T00:53:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1354793984 bytes 2024-09-15T00:53:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1430027264 bytes 2024-09-15T00:53:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1460250624 bytes 2024-09-15T00:53:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1511314432 bytes 2024-09-15T00:53:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1564173312 bytes 2024-09-15T00:53:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1577718784 bytes 2024-09-15T00:53:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1613560832 bytes 2024-09-15T00:53:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1632925696 bytes 2024-09-15T00:53:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1691542528 bytes 2024-09-15T00:53:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1777874944 bytes 2024-09-15T00:53:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1854834688 bytes 2024-09-15T00:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1888321536 bytes 2024-09-15T00:53:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1957054464 bytes 2024-09-15T00:53:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1979917312 bytes 2024-09-15T00:53:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2051340288 bytes 2024-09-15T00:53:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2072476672 bytes 2024-09-15T00:53:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2111977472 bytes 2024-09-15T00:53:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2165218304 bytes 2024-09-15T00:53:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2233476096 bytes 2024-09-15T00:53:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2263528448 bytes 2024-09-15T00:53:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2280495104 bytes 2024-09-15T00:53:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2305197056 bytes 2024-09-15T00:53:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_40 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T00:53:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:53:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:53:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2024-09-15T00:53:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:53:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:53:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:53:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-15T00:53:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T00:53:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:53:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-15T00:53:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:53:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-15T00:53:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_40': 2024-09-15T00:53:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:53:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:53:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 145b89db1450 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.145b89db1450 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:53:30 2024 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:53:30 2024 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:53:30 2024 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: dae82d23-f158-4ac8-9d92-19f4212caa20 2024-09-15T00:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:53:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:53:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-15T00:53:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T00:53:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=7 2024-09-15T00:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:53:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 13806592 bytes 2024-09-15T00:53:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 74537984 bytes 2024-09-15T00:53:32.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 102830080 bytes 2024-09-15T00:53:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 159136768 bytes 2024-09-15T00:53:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 177731584 bytes 2024-09-15T00:53:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 240326656 bytes 2024-09-15T00:53:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 292901888 bytes 2024-09-15T00:53:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 335617024 bytes 2024-09-15T00:53:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 396339200 bytes 2024-09-15T00:53:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 481955840 bytes 2024-09-15T00:53:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 514534400 bytes 2024-09-15T00:53:33.244 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:53:33.241+0000 7fc49cacc640 -1 reset not still connected to 0x560c54a27110 2024-09-15T00:53:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 566012928 bytes 2024-09-15T00:53:33.245 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:53:33.241+0000 7fc49cacc640 -1 reset not still connected to 0x560c54ac3ba0 2024-09-15T00:53:33.245 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:53:33.241+0000 7fc49cacc640 -1 reset not still connected to 0x560c59f8b6c0 2024-09-15T00:53:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 640392192 bytes 2024-09-15T00:53:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 704176128 bytes 2024-09-15T00:53:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 759003136 bytes 2024-09-15T00:53:33.837 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9236460428847567) > weight (0.25) 2024-09-15T00:53:33.837 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-15T00:53:33.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-15T00:53:33.838 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-15T00:53:33.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T00:53:33.836+0000 7fceba41d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 42157) UID: 1000 2024-09-15T00:53:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 825729024 bytes 2024-09-15T00:53:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 867651584 bytes 2024-09-15T00:53:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 895921152 bytes 2024-09-15T00:53:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 978003968 bytes 2024-09-15T00:53:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1082539008 bytes 2024-09-15T00:53:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1203919872 bytes 2024-09-15T00:53:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1233233920 bytes 2024-09-15T00:53:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1259265024 bytes 2024-09-15T00:53:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1342693376 bytes 2024-09-15T00:53:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1396031488 bytes 2024-09-15T00:53:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1412771840 bytes 2024-09-15T00:53:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1425103872 bytes 2024-09-15T00:53:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1456717824 bytes 2024-09-15T00:53:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1488414720 bytes 2024-09-15T00:53:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1498946560 bytes 2024-09-15T00:53:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1519965184 bytes 2024-09-15T00:53:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1575417856 bytes 2024-09-15T00:53:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1628974080 bytes 2024-09-15T00:53:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1695158272 bytes 2024-09-15T00:53:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1726283776 bytes 2024-09-15T00:53:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1758208000 bytes 2024-09-15T00:53:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1799340032 bytes 2024-09-15T00:53:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1818380288 bytes 2024-09-15T00:53:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1876118528 bytes 2024-09-15T00:53:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1885118464 bytes 2024-09-15T00:53:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1897964544 bytes 2024-09-15T00:53:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1898585088 bytes 2024-09-15T00:53:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1945041920 bytes 2024-09-15T00:53:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2002128896 bytes 2024-09-15T00:53:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2022381568 bytes 2024-09-15T00:53:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2110097408 bytes 2024-09-15T00:53:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2147693568 bytes 2024-09-15T00:53:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2172651520 bytes 2024-09-15T00:53:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2194663424 bytes 2024-09-15T00:53:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2252621824 bytes 2024-09-15T00:53:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2321981440 bytes 2024-09-15T00:53:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2345981952 bytes 2024-09-15T00:53:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2371090432 bytes 2024-09-15T00:53:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:53:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:53:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:53:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:53:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_41 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-15T00:53:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T00:53:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2024-09-15T00:53:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:53:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:53:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:53:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-15T00:53:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T00:53:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:53:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-15T00:53:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:53:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-15T00:53:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_41': 2024-09-15T00:53:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:53:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:53:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:53:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1467ad7012cd 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1467ad7012cd 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:53:38 2024 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:53:38 2024 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:53:38 2024 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: d8226719-7493-4a0d-a661-b9bbd9dfb0fa 2024-09-15T00:53:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:53:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:53:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-15T00:53:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T00:53:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:53:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=9 2024-09-15T00:53:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:53:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:53:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 55336960 bytes 2024-09-15T00:53:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 131460096 bytes 2024-09-15T00:53:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 189784064 bytes 2024-09-15T00:53:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 202248192 bytes 2024-09-15T00:53:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 253981696 bytes 2024-09-15T00:53:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 307651584 bytes 2024-09-15T00:53:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 317362176 bytes 2024-09-15T00:53:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 318350336 bytes 2024-09-15T00:53:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 344890368 bytes 2024-09-15T00:53:40.839 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-15T00:53:40.839 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:53:40.840 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-15T00:53:40.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-15T00:53:40.840 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-15T00:53:40.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-15T00:53:40.843 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-09-15T00:53:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 351685632 bytes 2024-09-15T00:53:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 352320512 bytes 2024-09-15T00:53:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 389886976 bytes 2024-09-15T00:53:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 461403136 bytes 2024-09-15T00:53:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 513684480 bytes 2024-09-15T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 523455488 bytes 2024-09-15T00:53:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 595200000 bytes 2024-09-15T00:53:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 638643200 bytes 2024-09-15T00:53:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 674169856 bytes 2024-09-15T00:53:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 733067264 bytes 2024-09-15T00:53:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 809741312 bytes 2024-09-15T00:53:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 831490048 bytes 2024-09-15T00:53:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 909409280 bytes 2024-09-15T00:53:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 959366144 bytes 2024-09-15T00:53:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1016230912 bytes 2024-09-15T00:53:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1056796672 bytes 2024-09-15T00:53:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1090001920 bytes 2024-09-15T00:53:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1115449344 bytes 2024-09-15T00:53:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1154096128 bytes 2024-09-15T00:53:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1218180096 bytes 2024-09-15T00:53:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1258007552 bytes 2024-09-15T00:53:43.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1304620032 bytes 2024-09-15T00:53:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1356812288 bytes 2024-09-15T00:53:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1358201856 bytes 2024-09-15T00:53:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1434474496 bytes 2024-09-15T00:53:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1541074944 bytes 2024-09-15T00:53:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1607270400 bytes 2024-09-15T00:53:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1621978112 bytes 2024-09-15T00:53:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1646854144 bytes 2024-09-15T00:53:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1686972416 bytes 2024-09-15T00:53:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1718969344 bytes 2024-09-15T00:53:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1724911616 bytes 2024-09-15T00:53:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1840394240 bytes 2024-09-15T00:53:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1900893184 bytes 2024-09-15T00:53:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1939691520 bytes 2024-09-15T00:53:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1953190912 bytes 2024-09-15T00:53:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1972319232 bytes 2024-09-15T00:53:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2012207104 bytes 2024-09-15T00:53:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2012843008 bytes 2024-09-15T00:53:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2085776384 bytes 2024-09-15T00:53:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2108729344 bytes 2024-09-15T00:53:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2133730304 bytes 2024-09-15T00:53:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2195188736 bytes 2024-09-15T00:53:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2244525056 bytes 2024-09-15T00:53:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2272629760 bytes 2024-09-15T00:53:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2291496960 bytes 2024-09-15T00:53:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2309950464 bytes 2024-09-15T00:53:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2322569216 bytes 2024-09-15T00:53:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2355497984 bytes 2024-09-15T00:53:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2394336256 bytes 2024-09-15T00:53:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2432940032 bytes 2024-09-15T00:53:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2477029376 bytes 2024-09-15T00:53:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2513642496 bytes 2024-09-15T00:53:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2544377856 bytes 2024-09-15T00:53:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2647056384 bytes 2024-09-15T00:53:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2703240192 bytes 2024-09-15T00:53:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2757417984 bytes 2024-09-15T00:53:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2766775296 bytes 2024-09-15T00:53:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2780253184 bytes 2024-09-15T00:53:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2824942592 bytes 2024-09-15T00:53:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_42 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:53:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2024-09-15T00:53:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:53:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:53:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:53:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-15T00:53:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T00:53:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:53:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-15T00:53:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:53:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-15T00:53:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_42': 2024-09-15T00:53:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:53:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:53:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:53:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1478a400ddb8 2024-09-15T00:53:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1478a400ddb8 2024-09-15T00:53:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:53:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:53:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:53:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:53:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:53:48 2024 2024-09-15T00:53:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:53:48 2024 2024-09-15T00:53:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:53:48 2024 2024-09-15T00:53:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:53:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:53:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: c0403aa6-7690-4fb7-abc5-df8d17c3bf1b 2024-09-15T00:53:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:53:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:53:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-15T00:53:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:53:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:53:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T00:53:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:53:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=13 2024-09-15T00:53:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:53:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:53:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 11073536 bytes 2024-09-15T00:53:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 34283520 bytes 2024-09-15T00:53:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 75572224 bytes 2024-09-15T00:53:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 164167680 bytes 2024-09-15T00:53:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 226805760 bytes 2024-09-15T00:53:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 262208512 bytes 2024-09-15T00:53:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 317559808 bytes 2024-09-15T00:53:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 372416512 bytes 2024-09-15T00:53:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 453792768 bytes 2024-09-15T00:53:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 492108800 bytes 2024-09-15T00:53:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 538490880 bytes 2024-09-15T00:53:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 550218752 bytes 2024-09-15T00:53:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 607145984 bytes 2024-09-15T00:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 635656192 bytes 2024-09-15T00:53:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 709167104 bytes 2024-09-15T00:53:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 724840448 bytes 2024-09-15T00:53:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 798026752 bytes 2024-09-15T00:53:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 862777344 bytes 2024-09-15T00:53:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 890151936 bytes 2024-09-15T00:53:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 979594240 bytes 2024-09-15T00:53:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1017149440 bytes 2024-09-15T00:53:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1022474240 bytes 2024-09-15T00:53:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1023115264 bytes 2024-09-15T00:53:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1047451648 bytes 2024-09-15T00:53:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1069696000 bytes 2024-09-15T00:53:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1115291648 bytes 2024-09-15T00:53:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1166737408 bytes 2024-09-15T00:53:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1195479040 bytes 2024-09-15T00:53:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1241330688 bytes 2024-09-15T00:53:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1275029504 bytes 2024-09-15T00:53:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1306983424 bytes 2024-09-15T00:53:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1337982976 bytes 2024-09-15T00:53:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1392836608 bytes 2024-09-15T00:53:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1431214080 bytes 2024-09-15T00:53:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1459002368 bytes 2024-09-15T00:53:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1495684096 bytes 2024-09-15T00:53:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1531637760 bytes 2024-09-15T00:53:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1552555008 bytes 2024-09-15T00:53:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1563811840 bytes 2024-09-15T00:53:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1590845440 bytes 2024-09-15T00:53:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1631302656 bytes 2024-09-15T00:53:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1672056832 bytes 2024-09-15T00:53:55.047 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-15T00:53:55.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-15T00:53:55.048 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-15T00:53:55.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T00:53:55.047+0000 7f2efd83a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 43917) UID: 1000 2024-09-15T00:53:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1730699264 bytes 2024-09-15T00:53:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1808353280 bytes 2024-09-15T00:53:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1827359744 bytes 2024-09-15T00:53:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1861272576 bytes 2024-09-15T00:53:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1951193088 bytes 2024-09-15T00:53:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2010070016 bytes 2024-09-15T00:53:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2076475392 bytes 2024-09-15T00:53:56.049 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-15T00:53:56.049 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:53:56.049 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-15T00:53:56.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-15T00:53:56.049 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-15T00:53:56.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-15T00:53:56.052 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-09-15T00:53:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2143761408 bytes 2024-09-15T00:53:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2174363648 bytes 2024-09-15T00:53:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2207170560 bytes 2024-09-15T00:53:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2232763392 bytes 2024-09-15T00:53:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2266435584 bytes 2024-09-15T00:53:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2292780032 bytes 2024-09-15T00:53:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2325668864 bytes 2024-09-15T00:53:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2396120064 bytes 2024-09-15T00:53:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2412718080 bytes 2024-09-15T00:53:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2472130560 bytes 2024-09-15T00:53:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2515243008 bytes 2024-09-15T00:53:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2515864576 bytes 2024-09-15T00:53:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2544259072 bytes 2024-09-15T00:53:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2613428224 bytes 2024-09-15T00:53:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2684894208 bytes 2024-09-15T00:53:58.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2711333888 bytes 2024-09-15T00:53:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2791122944 bytes 2024-09-15T00:53:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2878088192 bytes 2024-09-15T00:53:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2883954688 bytes 2024-09-15T00:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2921714688 bytes 2024-09-15T00:53:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2927088640 bytes 2024-09-15T00:53:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2955922432 bytes 2024-09-15T00:53:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3003791360 bytes 2024-09-15T00:53:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3034131456 bytes 2024-09-15T00:53:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3096427520 bytes 2024-09-15T00:53:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3148492800 bytes 2024-09-15T00:53:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3153668096 bytes 2024-09-15T00:53:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3216552960 bytes 2024-09-15T00:53:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3232583680 bytes 2024-09-15T00:53:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3233208320 bytes 2024-09-15T00:53:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3300074496 bytes 2024-09-15T00:54:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3404393472 bytes 2024-09-15T00:54:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3490528256 bytes 2024-09-15T00:54:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3536095232 bytes 2024-09-15T00:54:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3560610816 bytes 2024-09-15T00:54:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3611913216 bytes 2024-09-15T00:54:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3648923648 bytes 2024-09-15T00:54:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3704776704 bytes 2024-09-15T00:54:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3790647296 bytes 2024-09-15T00:54:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3817578496 bytes 2024-09-15T00:54:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3829703680 bytes 2024-09-15T00:54:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3914806272 bytes 2024-09-15T00:54:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3983792128 bytes 2024-09-15T00:54:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 4029293568 bytes 2024-09-15T00:54:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4029888512 bytes 2024-09-15T00:54:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4048430080 bytes 2024-09-15T00:54:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4111341568 bytes 2024-09-15T00:54:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4165248000 bytes 2024-09-15T00:54:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4244201472 bytes 2024-09-15T00:54:02.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4328465408 bytes 2024-09-15T00:54:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_43 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T00:54:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:54:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:54:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:54:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2024-09-15T00:54:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:54:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:54:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-15T00:54:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T00:54:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-15T00:54:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:54:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_43': 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 148df86ed43d 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.148df86ed43d 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:02 2024 2024-09-15T00:54:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:02 2024 2024-09-15T00:54:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:02 2024 2024-09-15T00:54:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:54:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:54:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: d65182ef-5444-4d87-9ce1-616af2d50d7b 2024-09-15T00:54:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:54:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:54:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-15T00:54:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T00:54:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:54:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=9 2024-09-15T00:54:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:54:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:54:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 111827968 bytes 2024-09-15T00:54:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 186915840 bytes 2024-09-15T00:54:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 230099968 bytes 2024-09-15T00:54:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 284423168 bytes 2024-09-15T00:54:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 303067136 bytes 2024-09-15T00:54:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 327859200 bytes 2024-09-15T00:54:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 369670144 bytes 2024-09-15T00:54:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 432550912 bytes 2024-09-15T00:54:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 487553024 bytes 2024-09-15T00:54:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 551494656 bytes 2024-09-15T00:54:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 580532224 bytes 2024-09-15T00:54:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 596720640 bytes 2024-09-15T00:54:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 675318784 bytes 2024-09-15T00:54:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 692383744 bytes 2024-09-15T00:54:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 741163008 bytes 2024-09-15T00:54:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 776856576 bytes 2024-09-15T00:54:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 841499648 bytes 2024-09-15T00:54:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 928899072 bytes 2024-09-15T00:54:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 940336128 bytes 2024-09-15T00:54:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 952809472 bytes 2024-09-15T00:54:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 990513152 bytes 2024-09-15T00:54:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1018546176 bytes 2024-09-15T00:54:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1031304192 bytes 2024-09-15T00:54:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1068625920 bytes 2024-09-15T00:54:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1113141248 bytes 2024-09-15T00:54:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1181597696 bytes 2024-09-15T00:54:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1224978432 bytes 2024-09-15T00:54:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1300759552 bytes 2024-09-15T00:54:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1325961216 bytes 2024-09-15T00:54:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1344136192 bytes 2024-09-15T00:54:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1398592512 bytes 2024-09-15T00:54:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1467497472 bytes 2024-09-15T00:54:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1486927872 bytes 2024-09-15T00:54:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1520839680 bytes 2024-09-15T00:54:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1556487168 bytes 2024-09-15T00:54:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1609668608 bytes 2024-09-15T00:54:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1665867776 bytes 2024-09-15T00:54:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1719177216 bytes 2024-09-15T00:54:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1725383680 bytes 2024-09-15T00:54:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1789699072 bytes 2024-09-15T00:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1851250688 bytes 2024-09-15T00:54:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1934678016 bytes 2024-09-15T00:54:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1942098944 bytes 2024-09-15T00:54:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2008168448 bytes 2024-09-15T00:54:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2082675712 bytes 2024-09-15T00:54:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2091490304 bytes 2024-09-15T00:54:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2096542720 bytes 2024-09-15T00:54:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2105710592 bytes 2024-09-15T00:54:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2149058560 bytes 2024-09-15T00:54:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2171882496 bytes 2024-09-15T00:54:10.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2172503040 bytes 2024-09-15T00:54:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2214242304 bytes 2024-09-15T00:54:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2239482880 bytes 2024-09-15T00:54:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2264913920 bytes 2024-09-15T00:54:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2330285056 bytes 2024-09-15T00:54:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2375214080 bytes 2024-09-15T00:54:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2435516416 bytes 2024-09-15T00:54:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2477295616 bytes 2024-09-15T00:54:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2503447552 bytes 2024-09-15T00:54:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2528385024 bytes 2024-09-15T00:54:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2586755072 bytes 2024-09-15T00:54:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2605940736 bytes 2024-09-15T00:54:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2651087872 bytes 2024-09-15T00:54:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2704824320 bytes 2024-09-15T00:54:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2749240320 bytes 2024-09-15T00:54:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2806601728 bytes 2024-09-15T00:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2807242752 bytes 2024-09-15T00:54:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2807860224 bytes 2024-09-15T00:54:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2864883712 bytes 2024-09-15T00:54:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2898113536 bytes 2024-09-15T00:54:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2964372480 bytes 2024-09-15T00:54:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_44 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T00:54:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:54:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-15T00:54:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T00:54:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:54:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:54:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:54:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2024-09-15T00:54:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:54:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:54:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-15T00:54:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T00:54:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-15T00:54:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:54:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-15T00:54:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_44': 2024-09-15T00:54:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:54:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:54:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1499b6e4d4aa 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1499b6e4d4aa 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:12 2024 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:12 2024 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:12 2024 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:54:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: ba608aff-65b4-4bab-bcb7-e9adf2c42922 2024-09-15T00:54:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:54:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:54:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-15T00:54:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T00:54:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:54:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=5 2024-09-15T00:54:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:54:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:54:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 25499648 bytes 2024-09-15T00:54:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 52464640 bytes 2024-09-15T00:54:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 79004672 bytes 2024-09-15T00:54:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 118751232 bytes 2024-09-15T00:54:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 175626240 bytes 2024-09-15T00:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 238589952 bytes 2024-09-15T00:54:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 285892608 bytes 2024-09-15T00:54:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 334679040 bytes 2024-09-15T00:54:15.113 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.112+0000 7fc49cacc640 -1 reset not still connected to 0x560c4f691110 2024-09-15T00:54:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 351111168 bytes 2024-09-15T00:54:15.123 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.121+0000 7fc49cacc640 -1 reset not still connected to 0x560c53e39040 2024-09-15T00:54:15.123 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.121+0000 7fc49cacc640 -1 reset not still connected to 0x560c573aa410 2024-09-15T00:54:15.123 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.121+0000 7fc49cacc640 -1 reset not still connected to 0x560c58b9c1a0 2024-09-15T00:54:15.126 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.124+0000 7fc49cacc640 -1 reset not still connected to 0x560c59eaed00 2024-09-15T00:54:15.132 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c5a88a410 2024-09-15T00:54:15.132 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c5c381860 2024-09-15T00:54:15.132 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c5d7e7110 2024-09-15T00:54:15.133 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c5f7da410 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c5fea91e0 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c611c9a00 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c62eb11e0 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c62ec95f0 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c63abbe10 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c64b185b0 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c66b06340 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c6bbd3380 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c6c7ced00 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c4f691110 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c53e39040 2024-09-15T00:54:15.134 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c5c381860 2024-09-15T00:54:15.135 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:15.130+0000 7fc49cacc640 -1 reset not still connected to 0x560c62ec95f0 2024-09-15T00:54:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 398190592 bytes 2024-09-15T00:54:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 401144832 bytes 2024-09-15T00:54:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 424424448 bytes 2024-09-15T00:54:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 434160640 bytes 2024-09-15T00:54:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 468654080 bytes 2024-09-15T00:54:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 506393600 bytes 2024-09-15T00:54:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 535497728 bytes 2024-09-15T00:54:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 562834432 bytes 2024-09-15T00:54:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 591504384 bytes 2024-09-15T00:54:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 670982144 bytes 2024-09-15T00:54:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 780610560 bytes 2024-09-15T00:54:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 849876992 bytes 2024-09-15T00:54:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 913169408 bytes 2024-09-15T00:54:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 949660672 bytes 2024-09-15T00:54:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1029228544 bytes 2024-09-15T00:54:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1043571712 bytes 2024-09-15T00:54:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1089824768 bytes 2024-09-15T00:54:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1105687552 bytes 2024-09-15T00:54:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1117114368 bytes 2024-09-15T00:54:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1162720256 bytes 2024-09-15T00:54:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1199789056 bytes 2024-09-15T00:54:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1273047040 bytes 2024-09-15T00:54:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1291468800 bytes 2024-09-15T00:54:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1337314304 bytes 2024-09-15T00:54:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1338085376 bytes 2024-09-15T00:54:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1386943488 bytes 2024-09-15T00:54:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1422149632 bytes 2024-09-15T00:54:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1493163008 bytes 2024-09-15T00:54:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1525804032 bytes 2024-09-15T00:54:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:54:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:54:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:54:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:54:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_45 2024-09-15T00:54:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-15T00:54:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T00:54:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:54:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:54:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:54:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2024-09-15T00:54:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:54:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:54:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-15T00:54:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T00:54:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-15T00:54:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:54:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-15T00:54:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_45': 2024-09-15T00:54:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:54:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:54:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:54:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 14a5a3fbbb9 2024-09-15T00:54:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.14a5a3fbbb9 2024-09-15T00:54:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:54:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:54:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:54:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:54:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:18 2024 2024-09-15T00:54:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:18 2024 2024-09-15T00:54:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:18 2024 2024-09-15T00:54:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:54:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:54:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 7c98f381-5ad1-4de6-abca-0422679de2a8 2024-09-15T00:54:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:54:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:54:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-15T00:54:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T00:54:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:54:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=11 2024-09-15T00:54:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:54:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:54:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 70627328 bytes 2024-09-15T00:54:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 95348736 bytes 2024-09-15T00:54:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 170222592 bytes 2024-09-15T00:54:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 235169792 bytes 2024-09-15T00:54:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 276388864 bytes 2024-09-15T00:54:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 324643840 bytes 2024-09-15T00:54:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 332122112 bytes 2024-09-15T00:54:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 379456512 bytes 2024-09-15T00:54:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 427685888 bytes 2024-09-15T00:54:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 513666048 bytes 2024-09-15T00:54:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 582020096 bytes 2024-09-15T00:54:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 630170624 bytes 2024-09-15T00:54:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 682036224 bytes 2024-09-15T00:54:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 739057664 bytes 2024-09-15T00:54:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 781992960 bytes 2024-09-15T00:54:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 827590656 bytes 2024-09-15T00:54:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 898410496 bytes 2024-09-15T00:54:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 944675840 bytes 2024-09-15T00:54:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 988055552 bytes 2024-09-15T00:54:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1051737088 bytes 2024-09-15T00:54:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1119546368 bytes 2024-09-15T00:54:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1144603648 bytes 2024-09-15T00:54:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1191182336 bytes 2024-09-15T00:54:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1219215360 bytes 2024-09-15T00:54:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1293479936 bytes 2024-09-15T00:54:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1331818496 bytes 2024-09-15T00:54:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1384344576 bytes 2024-09-15T00:54:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1416052736 bytes 2024-09-15T00:54:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1481939968 bytes 2024-09-15T00:54:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1523000320 bytes 2024-09-15T00:54:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1541961728 bytes 2024-09-15T00:54:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1596576768 bytes 2024-09-15T00:54:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1654130688 bytes 2024-09-15T00:54:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1735011328 bytes 2024-09-15T00:54:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1791645696 bytes 2024-09-15T00:54:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1839823872 bytes 2024-09-15T00:54:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1905825792 bytes 2024-09-15T00:54:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1974616064 bytes 2024-09-15T00:54:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1997643776 bytes 2024-09-15T00:54:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 2042512384 bytes 2024-09-15T00:54:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 2047816704 bytes 2024-09-15T00:54:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 2122388480 bytes 2024-09-15T00:54:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2153307136 bytes 2024-09-15T00:54:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2188514304 bytes 2024-09-15T00:54:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2298530816 bytes 2024-09-15T00:54:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2361349120 bytes 2024-09-15T00:54:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2411491328 bytes 2024-09-15T00:54:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2481180672 bytes 2024-09-15T00:54:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2550549504 bytes 2024-09-15T00:54:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2571333632 bytes 2024-09-15T00:54:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2598614016 bytes 2024-09-15T00:54:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2632807424 bytes 2024-09-15T00:54:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2748212224 bytes 2024-09-15T00:54:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2801146880 bytes 2024-09-15T00:54:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2826743808 bytes 2024-09-15T00:54:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2868276224 bytes 2024-09-15T00:54:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2902519808 bytes 2024-09-15T00:54:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2922444800 bytes 2024-09-15T00:54:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2934516736 bytes 2024-09-15T00:54:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2947053568 bytes 2024-09-15T00:54:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2957299712 bytes 2024-09-15T00:54:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 3008444416 bytes 2024-09-15T00:54:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 3038993408 bytes 2024-09-15T00:54:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 3121931264 bytes 2024-09-15T00:54:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 3165838336 bytes 2024-09-15T00:54:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 3226323968 bytes 2024-09-15T00:54:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3272429568 bytes 2024-09-15T00:54:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3291422720 bytes 2024-09-15T00:54:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3360482304 bytes 2024-09-15T00:54:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3406963712 bytes 2024-09-15T00:54:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3427109888 bytes 2024-09-15T00:54:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3466729472 bytes 2024-09-15T00:54:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3507531776 bytes 2024-09-15T00:54:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3529944064 bytes 2024-09-15T00:54:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3552526336 bytes 2024-09-15T00:54:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3597467648 bytes 2024-09-15T00:54:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3612909568 bytes 2024-09-15T00:54:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3652894720 bytes 2024-09-15T00:54:30.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3662088192 bytes 2024-09-15T00:54:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3678182400 bytes 2024-09-15T00:54:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3764299776 bytes 2024-09-15T00:54:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3798197248 bytes 2024-09-15T00:54:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3850909696 bytes 2024-09-15T00:54:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:54:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:54:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:54:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:54:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_46 2024-09-15T00:54:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-15T00:54:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T00:54:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:54:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:54:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:54:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2024-09-15T00:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-15T00:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T00:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-15T00:54:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:54:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_46': 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 14b189f8ada6 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.14b189f8ada6 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:30 2024 2024-09-15T00:54:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:30 2024 2024-09-15T00:54:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:30 2024 2024-09-15T00:54:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:54:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:54:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 6a06876f-8afd-4b45-82aa-fe83ff9bc561 2024-09-15T00:54:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:54:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:54:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-15T00:54:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T00:54:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:54:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=11 2024-09-15T00:54:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:54:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 66618368 bytes 2024-09-15T00:54:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 111027200 bytes 2024-09-15T00:54:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 111623168 bytes 2024-09-15T00:54:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 159200256 bytes 2024-09-15T00:54:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 191428608 bytes 2024-09-15T00:54:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 266784768 bytes 2024-09-15T00:54:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 274049024 bytes 2024-09-15T00:54:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 319446016 bytes 2024-09-15T00:54:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 361403392 bytes 2024-09-15T00:54:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 427242496 bytes 2024-09-15T00:54:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 448522240 bytes 2024-09-15T00:54:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 545064960 bytes 2024-09-15T00:54:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 604227584 bytes 2024-09-15T00:54:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 633537536 bytes 2024-09-15T00:54:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 634156032 bytes 2024-09-15T00:54:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 705132544 bytes 2024-09-15T00:54:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 726062080 bytes 2024-09-15T00:54:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 755447808 bytes 2024-09-15T00:54:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 771911680 bytes 2024-09-15T00:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 772545536 bytes 2024-09-15T00:54:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 825313280 bytes 2024-09-15T00:54:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 906604544 bytes 2024-09-15T00:54:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 924195840 bytes 2024-09-15T00:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 924795904 bytes 2024-09-15T00:54:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 969669632 bytes 2024-09-15T00:54:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 978785280 bytes 2024-09-15T00:54:35.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1005143040 bytes 2024-09-15T00:54:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1025380352 bytes 2024-09-15T00:54:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1046173696 bytes 2024-09-15T00:54:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1119756288 bytes 2024-09-15T00:54:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1152840704 bytes 2024-09-15T00:54:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1187427328 bytes 2024-09-15T00:54:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1286577152 bytes 2024-09-15T00:54:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1308134400 bytes 2024-09-15T00:54:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1357970432 bytes 2024-09-15T00:54:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1381399552 bytes 2024-09-15T00:54:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1382030336 bytes 2024-09-15T00:54:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1394271232 bytes 2024-09-15T00:54:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1431089152 bytes 2024-09-15T00:54:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1447483392 bytes 2024-09-15T00:54:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1542211584 bytes 2024-09-15T00:54:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1609253888 bytes 2024-09-15T00:54:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1656498176 bytes 2024-09-15T00:54:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1671926784 bytes 2024-09-15T00:54:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1701659648 bytes 2024-09-15T00:54:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1733681152 bytes 2024-09-15T00:54:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1750600704 bytes 2024-09-15T00:54:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1802514432 bytes 2024-09-15T00:54:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1838038016 bytes 2024-09-15T00:54:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1838629888 bytes 2024-09-15T00:54:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1840875520 bytes 2024-09-15T00:54:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 1856796672 bytes 2024-09-15T00:54:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 1888310272 bytes 2024-09-15T00:54:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 1961613312 bytes 2024-09-15T00:54:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2011054080 bytes 2024-09-15T00:54:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2037955584 bytes 2024-09-15T00:54:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2102140928 bytes 2024-09-15T00:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2168962048 bytes 2024-09-15T00:54:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2205110272 bytes 2024-09-15T00:54:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2277527552 bytes 2024-09-15T00:54:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2384268288 bytes 2024-09-15T00:54:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2418062336 bytes 2024-09-15T00:54:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2418677760 bytes 2024-09-15T00:54:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2508196864 bytes 2024-09-15T00:54:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2527295488 bytes 2024-09-15T00:54:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2546447360 bytes 2024-09-15T00:54:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2653110272 bytes 2024-09-15T00:54:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2689426432 bytes 2024-09-15T00:54:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2750585856 bytes 2024-09-15T00:54:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2807888896 bytes 2024-09-15T00:54:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2842685440 bytes 2024-09-15T00:54:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2869613568 bytes 2024-09-15T00:54:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2897314816 bytes 2024-09-15T00:54:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2924649472 bytes 2024-09-15T00:54:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2956774400 bytes 2024-09-15T00:54:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2980619264 bytes 2024-09-15T00:54:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3009843200 bytes 2024-09-15T00:54:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3062194176 bytes 2024-09-15T00:54:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3102300160 bytes 2024-09-15T00:54:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3140604928 bytes 2024-09-15T00:54:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3210891264 bytes 2024-09-15T00:54:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3293820928 bytes 2024-09-15T00:54:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3326775296 bytes 2024-09-15T00:54:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3361267712 bytes 2024-09-15T00:54:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3394851840 bytes 2024-09-15T00:54:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:54:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:54:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_47 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:54:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-15T00:54:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T00:54:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:54:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:54:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:54:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2024-09-15T00:54:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:54:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:54:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-15T00:54:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T00:54:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-15T00:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:54:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-15T00:54:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_47': 2024-09-15T00:54:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:54:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:54:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:54:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 14bde3714e01 2024-09-15T00:54:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.14bde3714e01 2024-09-15T00:54:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:54:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:54:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:54:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:54:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:42 2024 2024-09-15T00:54:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:42 2024 2024-09-15T00:54:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:42 2024 2024-09-15T00:54:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:54:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:54:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 2f43fe62-b4f5-43a2-8094-d1a91f06de6f 2024-09-15T00:54:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:54:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:54:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-15T00:54:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T00:54:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:54:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=5 2024-09-15T00:54:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:54:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:54:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 92988416 bytes 2024-09-15T00:54:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 163722240 bytes 2024-09-15T00:54:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 182104064 bytes 2024-09-15T00:54:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 244410368 bytes 2024-09-15T00:54:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 285424640 bytes 2024-09-15T00:54:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 295327744 bytes 2024-09-15T00:54:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 381857792 bytes 2024-09-15T00:54:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 434158592 bytes 2024-09-15T00:54:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 488534016 bytes 2024-09-15T00:54:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 495580160 bytes 2024-09-15T00:54:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 496221184 bytes 2024-09-15T00:54:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 610800640 bytes 2024-09-15T00:54:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 611404800 bytes 2024-09-15T00:54:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 639269888 bytes 2024-09-15T00:54:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 673545216 bytes 2024-09-15T00:54:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 674141184 bytes 2024-09-15T00:54:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 683446272 bytes 2024-09-15T00:54:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 721430528 bytes 2024-09-15T00:54:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 787012608 bytes 2024-09-15T00:54:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 869322752 bytes 2024-09-15T00:54:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 899981312 bytes 2024-09-15T00:54:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 935890944 bytes 2024-09-15T00:54:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1006131200 bytes 2024-09-15T00:54:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1041849344 bytes 2024-09-15T00:54:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1115023360 bytes 2024-09-15T00:54:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1188043776 bytes 2024-09-15T00:54:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1214600192 bytes 2024-09-15T00:54:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1294067712 bytes 2024-09-15T00:54:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1332698112 bytes 2024-09-15T00:54:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1393629184 bytes 2024-09-15T00:54:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1448369152 bytes 2024-09-15T00:54:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1488872448 bytes 2024-09-15T00:54:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1518475264 bytes 2024-09-15T00:54:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1553675264 bytes 2024-09-15T00:54:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1586335744 bytes 2024-09-15T00:54:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1617319936 bytes 2024-09-15T00:54:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1632476160 bytes 2024-09-15T00:54:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:54:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:54:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_48 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2024-09-15T00:54:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:54:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:54:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-15T00:54:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T00:54:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-15T00:54:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:54:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-15T00:54:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_48': 2024-09-15T00:54:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:54:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:54:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:54:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 14c97b67e74a 2024-09-15T00:54:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.14c97b67e74a 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:48 2024 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:48 2024 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:48 2024 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 1bec5446-ddd9-4c25-9934-0781fdf9ac61 2024-09-15T00:54:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:54:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:54:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-15T00:54:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T00:54:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:54:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=5 2024-09-15T00:54:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:54:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:54:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 79429632 bytes 2024-09-15T00:54:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 119877632 bytes 2024-09-15T00:54:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 151653376 bytes 2024-09-15T00:54:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 233954304 bytes 2024-09-15T00:54:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 252009472 bytes 2024-09-15T00:54:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 356666368 bytes 2024-09-15T00:54:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 409609216 bytes 2024-09-15T00:54:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 421728256 bytes 2024-09-15T00:54:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 482544640 bytes 2024-09-15T00:54:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 515519488 bytes 2024-09-15T00:54:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 558888960 bytes 2024-09-15T00:54:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 599908352 bytes 2024-09-15T00:54:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 613635072 bytes 2024-09-15T00:54:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 677852160 bytes 2024-09-15T00:54:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 704437248 bytes 2024-09-15T00:54:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 761786368 bytes 2024-09-15T00:54:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 783025152 bytes 2024-09-15T00:54:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 843496448 bytes 2024-09-15T00:54:52.272 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:54:52.270+0000 7fc49cacc640 -1 reset not still connected to 0x560c66f90820 2024-09-15T00:54:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 882502656 bytes 2024-09-15T00:54:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 910262272 bytes 2024-09-15T00:54:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 959692800 bytes 2024-09-15T00:54:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1042842624 bytes 2024-09-15T00:54:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1064131584 bytes 2024-09-15T00:54:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1096845312 bytes 2024-09-15T00:54:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1153861632 bytes 2024-09-15T00:54:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1178238976 bytes 2024-09-15T00:54:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1242882048 bytes 2024-09-15T00:54:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1302563840 bytes 2024-09-15T00:54:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1335336960 bytes 2024-09-15T00:54:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1389810688 bytes 2024-09-15T00:54:53.844 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-15T00:54:53.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-15T00:54:53.844 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-15T00:54:53.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T00:54:53.843+0000 7fb4c016f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 48151) UID: 1000 2024-09-15T00:54:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1429345280 bytes 2024-09-15T00:54:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1466170368 bytes 2024-09-15T00:54:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1481794560 bytes 2024-09-15T00:54:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1515675648 bytes 2024-09-15T00:54:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1555484672 bytes 2024-09-15T00:54:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1590203392 bytes 2024-09-15T00:54:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1675912192 bytes 2024-09-15T00:54:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1733449728 bytes 2024-09-15T00:54:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:54:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:54:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:54:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:54:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_49 2024-09-15T00:54:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-15T00:54:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T00:54:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:54:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:54:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:54:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:54:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2024-09-15T00:54:54.845 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-15T00:54:54.846 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:54:54.846 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-15T00:54:54.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-15T00:54:54.847 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-15T00:54:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:54:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:54:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:54:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-15T00:54:54.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-15T00:54:54.850 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-15T00:54:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T00:54:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:54:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-15T00:54:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:54:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-15T00:54:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_49': 2024-09-15T00:54:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:54:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 14d424a46814 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.14d424a46814 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:54 2024 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:54 2024 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:54 2024 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 3ecb23d9-a459-4c26-b283-80a7a224cdfb 2024-09-15T00:54:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:54:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:54:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-15T00:54:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:54:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:54:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T00:54:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:54:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=7 2024-09-15T00:54:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:54:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:54:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 48151552 bytes 2024-09-15T00:54:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 146310144 bytes 2024-09-15T00:54:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 163794944 bytes 2024-09-15T00:54:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 164404224 bytes 2024-09-15T00:54:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 207078400 bytes 2024-09-15T00:54:56.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 207694848 bytes 2024-09-15T00:54:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 272350208 bytes 2024-09-15T00:54:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 314595328 bytes 2024-09-15T00:54:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 336920576 bytes 2024-09-15T00:54:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 348521472 bytes 2024-09-15T00:54:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 387277824 bytes 2024-09-15T00:54:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 407347200 bytes 2024-09-15T00:54:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 412792832 bytes 2024-09-15T00:54:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 477122560 bytes 2024-09-15T00:54:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 533371904 bytes 2024-09-15T00:54:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 591598592 bytes 2024-09-15T00:54:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 646720512 bytes 2024-09-15T00:54:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 720762880 bytes 2024-09-15T00:54:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 768989184 bytes 2024-09-15T00:54:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 810172416 bytes 2024-09-15T00:54:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 847546368 bytes 2024-09-15T00:54:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 858934272 bytes 2024-09-15T00:54:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 882763776 bytes 2024-09-15T00:54:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 959769600 bytes 2024-09-15T00:54:59.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 974781440 bytes 2024-09-15T00:54:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 976291840 bytes 2024-09-15T00:54:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 992927744 bytes 2024-09-15T00:54:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1017729024 bytes 2024-09-15T00:54:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1068184576 bytes 2024-09-15T00:54:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1079544832 bytes 2024-09-15T00:54:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1126606848 bytes 2024-09-15T00:55:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1221340160 bytes 2024-09-15T00:55:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1264580608 bytes 2024-09-15T00:55:00.317 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1266318336 bytes 2024-09-15T00:55:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1321116672 bytes 2024-09-15T00:55:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1387307008 bytes 2024-09-15T00:55:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1422351360 bytes 2024-09-15T00:55:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1462371328 bytes 2024-09-15T00:55:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1518909440 bytes 2024-09-15T00:55:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1607667712 bytes 2024-09-15T00:55:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1672694784 bytes 2024-09-15T00:55:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1721628672 bytes 2024-09-15T00:55:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1754183680 bytes 2024-09-15T00:55:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1785455616 bytes 2024-09-15T00:55:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1803151360 bytes 2024-09-15T00:55:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1878635520 bytes 2024-09-15T00:55:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1890095104 bytes 2024-09-15T00:55:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1937442816 bytes 2024-09-15T00:55:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1978465280 bytes 2024-09-15T00:55:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2030961664 bytes 2024-09-15T00:55:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2094937088 bytes 2024-09-15T00:55:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2152777728 bytes 2024-09-15T00:55:02.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2153374720 bytes 2024-09-15T00:55:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:55:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:55:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:55:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:55:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_50 2024-09-15T00:55:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-15T00:55:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:55:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:55:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T00:55:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:55:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:55:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n snapshot ']' 2024-09-15T00:55:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:55:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-15T00:55:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:55:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:55:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T00:55:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:55:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T00:55:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 128M ']' 2024-09-15T00:55:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=128M 2024-09-15T00:55:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T00:55:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2024-09-15T00:55:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ is_pool_mirror_mode_image mirror 2024-09-15T00:55:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' image = image ']' 2024-09-15T00:55:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-15T00:55:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T00:55:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T00:55:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-15T00:55:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T00:55:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_50': 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 14e5e63403b6 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.14e5e63403b6 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:55:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:55:02 2024 2024-09-15T00:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:55:02 2024 2024-09-15T00:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:55:02 2024 2024-09-15T00:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 5970ee5d-ad39-4b4f-828b-7d4f1adceca5 2024-09-15T00:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T00:55:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n '' ']' 2024-09-15T00:55:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-15T00:55:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T00:55:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T00:55:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=13 2024-09-15T00:55:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T00:55:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-15T00:55:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 33249280 bytes 2024-09-15T00:55:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 73663488 bytes 2024-09-15T00:55:04.243 INFO:tasks.ceph.cluster2.osd.1.smithi059.stderr:2024-09-15T00:55:04.240+0000 7f9966144640 -1 reset not still connected to 0x55ef67fae410 2024-09-15T00:55:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 99083264 bytes 2024-09-15T00:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 127073280 bytes 2024-09-15T00:55:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 163680256 bytes 2024-09-15T00:55:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 229676032 bytes 2024-09-15T00:55:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 292838400 bytes 2024-09-15T00:55:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 346301440 bytes 2024-09-15T00:55:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 390242304 bytes 2024-09-15T00:55:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 471454720 bytes 2024-09-15T00:55:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 476248064 bytes 2024-09-15T00:55:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 552845312 bytes 2024-09-15T00:55:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 590229504 bytes 2024-09-15T00:55:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 590854144 bytes 2024-09-15T00:55:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 630854656 bytes 2024-09-15T00:55:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 669611008 bytes 2024-09-15T00:55:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 712975360 bytes 2024-09-15T00:55:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 735563776 bytes 2024-09-15T00:55:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 780834816 bytes 2024-09-15T00:55:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 789252096 bytes 2024-09-15T00:55:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 803830784 bytes 2024-09-15T00:55:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 900423680 bytes 2024-09-15T00:55:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 992766976 bytes 2024-09-15T00:55:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1033330688 bytes 2024-09-15T00:55:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1059960832 bytes 2024-09-15T00:55:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1094501376 bytes 2024-09-15T00:55:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1159383040 bytes 2024-09-15T00:55:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1190438912 bytes 2024-09-15T00:55:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1217997824 bytes 2024-09-15T00:55:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1249399808 bytes 2024-09-15T00:55:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1255815168 bytes 2024-09-15T00:55:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1291494400 bytes 2024-09-15T00:55:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1304440832 bytes 2024-09-15T00:55:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1305060352 bytes 2024-09-15T00:55:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1350891520 bytes 2024-09-15T00:55:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1353862144 bytes 2024-09-15T00:55:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1418242048 bytes 2024-09-15T00:55:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1455027200 bytes 2024-09-15T00:55:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1500605440 bytes 2024-09-15T00:55:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1512990720 bytes 2024-09-15T00:55:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1534637056 bytes 2024-09-15T00:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1565462528 bytes 2024-09-15T00:55:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1637794816 bytes 2024-09-15T00:55:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1701782528 bytes 2024-09-15T00:55:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1702404096 bytes 2024-09-15T00:55:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1754896384 bytes 2024-09-15T00:55:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1786420224 bytes 2024-09-15T00:55:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1845996544 bytes 2024-09-15T00:55:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1883566080 bytes 2024-09-15T00:55:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1949347840 bytes 2024-09-15T00:55:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 1957627904 bytes 2024-09-15T00:55:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 1991554048 bytes 2024-09-15T00:55:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2040117248 bytes 2024-09-15T00:55:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2070359040 bytes 2024-09-15T00:55:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2105119744 bytes 2024-09-15T00:55:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2149864448 bytes 2024-09-15T00:55:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2212457472 bytes 2024-09-15T00:55:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2269913088 bytes 2024-09-15T00:55:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2317673472 bytes 2024-09-15T00:55:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2351962112 bytes 2024-09-15T00:55:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2403933184 bytes 2024-09-15T00:55:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2461814784 bytes 2024-09-15T00:55:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2515582976 bytes 2024-09-15T00:55:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2520608768 bytes 2024-09-15T00:55:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2535858176 bytes 2024-09-15T00:55:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2570460160 bytes 2024-09-15T00:55:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2611504128 bytes 2024-09-15T00:55:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2661472256 bytes 2024-09-15T00:55:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2673013760 bytes 2024-09-15T00:55:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2698414080 bytes 2024-09-15T00:55:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2716306432 bytes 2024-09-15T00:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2753369088 bytes 2024-09-15T00:55:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2775094272 bytes 2024-09-15T00:55:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2801739776 bytes 2024-09-15T00:55:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2850676736 bytes 2024-09-15T00:55:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2905648128 bytes 2024-09-15T00:55:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 2944746496 bytes 2024-09-15T00:55:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 2967360512 bytes 2024-09-15T00:55:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 2988567552 bytes 2024-09-15T00:55:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3006288896 bytes 2024-09-15T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3006911488 bytes 2024-09-15T00:55:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3079779328 bytes 2024-09-15T00:55:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3167125504 bytes 2024-09-15T00:55:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3207005184 bytes 2024-09-15T00:55:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3273867264 bytes 2024-09-15T00:55:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3345297408 bytes 2024-09-15T00:55:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3386653696 bytes 2024-09-15T00:55:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3429428224 bytes 2024-09-15T00:55:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3512001536 bytes 2024-09-15T00:55:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3606233088 bytes 2024-09-15T00:55:15.141 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:55:15.126+0000 7fc49cacc640 -1 reset not still connected to 0x560c6c7ced00 2024-09-15T00:55:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3638714368 bytes 2024-09-15T00:55:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3722688512 bytes 2024-09-15T00:55:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3754997760 bytes 2024-09-15T00:55:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3809434624 bytes 2024-09-15T00:55:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 3855234048 bytes 2024-09-15T00:55:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 3919309824 bytes 2024-09-15T00:55:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 3963256832 bytes 2024-09-15T00:55:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 3988974592 bytes 2024-09-15T00:55:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4035233792 bytes 2024-09-15T00:55:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4104005632 bytes 2024-09-15T00:55:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4164709376 bytes 2024-09-15T00:55:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4262749184 bytes 2024-09-15T00:55:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T00:55:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T00:55:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T00:55:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-15T00:55:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_count=50 2024-09-15T00:55:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:55:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local count 2024-09-15T00:55:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_count=0 2024-09-15T00:55:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:55:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 40 2024-09-15T00:55:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in `seq 1 40` 2024-09-15T00:55:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 1 -ne 1 2024-09-15T00:55:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-15T00:55:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep 'images: ' 2024-09-15T00:55:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cut '-d ' -f 2 2024-09-15T00:55:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: failed to query services: (13) Permission denied 2024-09-15T00:55:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ count=50 2024-09-15T00:55:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 50 = 50 2024-09-15T00:55:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-15T00:55:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:55:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state 2024-09-15T00:55:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 40 2024-09-15T00:55:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in `seq 1 40` 2024-09-15T00:55:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 1 -ne 1 2024-09-15T00:55:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-15T00:55:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep 'image health:' 2024-09-15T00:55:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cut '-d ' -f 3 2024-09-15T00:55:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: failed to query services: (13) Permission denied 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=OK 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test OK = ERROR 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test OK = OK 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_count=50 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local count 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_count=0 2024-09-15T00:55:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:55:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 40 2024-09-15T00:55:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in `seq 1 40` 2024-09-15T00:55:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 1 -ne 1 2024-09-15T00:55:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-15T00:55:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep 'images: ' 2024-09-15T00:55:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cut '-d ' -f 2 2024-09-15T00:55:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: failed to query services: (13) Permission denied 2024-09-15T00:55:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ count=50 2024-09-15T00:55:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 50 = 50 2024-09-15T00:55:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-15T00:55:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:55:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state 2024-09-15T00:55:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 40 2024-09-15T00:55:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in `seq 1 40` 2024-09-15T00:55:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 1 -ne 1 2024-09-15T00:55:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-15T00:55:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep 'image health:' 2024-09-15T00:55:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cut '-d ' -f 3 2024-09-15T00:55:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: failed to query services: (13) Permission denied 2024-09-15T00:55:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=WARNING 2024-09-15T00:55:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test WARNING = ERROR 2024-09-15T00:55:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test WARNING = OK 2024-09-15T00:55:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in `seq 1 40` 2024-09-15T00:55:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 2 -ne 1 2024-09-15T00:55:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 30 2024-09-15T00:55:42.053 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.69453511745143) > weight (0.25) 2024-09-15T00:55:42.054 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4032383506374103) > weight (0.25) 2024-09-15T00:55:42.054 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-15T00:55:42.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-15T00:55:42.055 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-15T00:55:42.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T00:55:42.054+0000 7f95e7d37640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 46591) UID: 1000 2024-09-15T00:55:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-15T00:55:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep 'image health:' 2024-09-15T00:55:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cut '-d ' -f 3 2024-09-15T00:55:47.055 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-15T00:55:47.056 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:55:47.056 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-15T00:55:47.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-15T00:55:47.056 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-15T00:55:47.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-15T00:55:47.059 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-09-15T00:55:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: failed to query services: (13) Permission denied 2024-09-15T00:55:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=OK 2024-09-15T00:55:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test OK = ERROR 2024-09-15T00:55:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test OK = OK 2024-09-15T00:55:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ testlog 'TEST: compare many images' 2024-09-15T00:55:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ tee -a /tmp/tmp.nS85ynkalX/rbd-mirror.test.log 2024-09-15T00:55:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ date '+%F %T' 2024-09-15T00:55:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo 2024-09-15 00:55:47 TEST: compare many images 2024-09-15T00:55:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:2024-09-15 00:55:47 TEST: compare many images 2024-09-15T00:55:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 50 2024-09-15T00:55:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:55:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_1 2024-09-15T00:55:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-15T00:55:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:55:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:55:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-15T00:55:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-15T00:55:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:55:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:55:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:55:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:55:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-15T00:55:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:55:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:55:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:55:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-15T00:55:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:55:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:55:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:55:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:55:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:55:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:55:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:55:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:55:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:55:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:55:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:55:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:55:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:55:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:55:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:55:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:55:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:55:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:55:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-15T00:55:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-15T00:55:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:55:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:55:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:55:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-15T00:55:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:55:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:55:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:55:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:55:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:55:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:55:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:55:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:55:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:55:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:55:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:55:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:55:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:55:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:55:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:55:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:55:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:55:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:55:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:55:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:55:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:55:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-15T00:55:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-15T00:55:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:55:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:55:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:55:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-09-15T00:55:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:55:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:55:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:55:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:55:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:55:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:55:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:55:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:55:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:55:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:55:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:55:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:55:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:55:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:55:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:55:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:55:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:55:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:55:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:55:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:55:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:55:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:55:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-09-15T00:55:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:55:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:55:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:55:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:55:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:55:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:55:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:55:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-15T00:55:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-15T00:55:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_1.status 2024-09-15T00:55:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:55:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_1.status 2024-09-15T00:55:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-15T00:55:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_1.status 2024-09-15T00:55:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:55:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.status 2024-09-15T00:55:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-15T00:55:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-15T00:55:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 86 2024-09-15T00:55:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_1.status 2024-09-15T00:55:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_1.status 2024-09-15T00:55:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-15T00:55:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:55:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:55:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=86 2024-09-15T00:55:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:55:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:55:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:55:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.status 2024-09-15T00:55:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.status 2024-09-15T00:55:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-15T00:55:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:55:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:55:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:55:51.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:55:51.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:55:51.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:55:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:55:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 86 = '' 2024-09-15T00:55:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:55:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:55:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.status 2024-09-15T00:55:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.status 2024-09-15T00:55:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-15T00:55:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:55:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:55:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:55:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:55:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:55:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:55:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:55:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 86 = '' 2024-09-15T00:55:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:55:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:55:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.status 2024-09-15T00:55:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.status 2024-09-15T00:55:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-15T00:55:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:55:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:55:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=86 2024-09-15T00:55:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 86 = 86 2024-09-15T00:55:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-15T00:55:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:55:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:55:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:55:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:55:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-15T00:55:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_1': 2024-09-15T00:55:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:55:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:55:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:55:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 118213d2486b 2024-09-15T00:55:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.118213d2486b 2024-09-15T00:55:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:55:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:55:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:55:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:55:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:46:38 2024 2024-09-15T00:55:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:46:38 2024 2024-09-15T00:55:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:46:38 2024 2024-09-15T00:55:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:55:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:55:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:55:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: f6dc2e78-22b9-4458-a5da-18ecc500097e 2024-09-15T00:55:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:55:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-15T00:55:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T00:55:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:55:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:55:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_1.export 2024-09-15T00:55:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.export 2024-09-15T00:55:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_1.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.export 2024-09-15T00:55:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_1.export 2024-09-15T00:55:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:55:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.export 2024-09-15T00:55:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:55:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_1.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.export 2024-09-15T00:55:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_1.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_1.export 2024-09-15T00:55:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:55:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_2 2024-09-15T00:55:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-15T00:55:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:55:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:55:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-15T00:55:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-15T00:55:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:55:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:55:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:55:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:55:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-15T00:55:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:55:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:55:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:55:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-15T00:55:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:55:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:55:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:55:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:55:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:55:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:55:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:55:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:55:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:55:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:55:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:55:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:55:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:55:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:55:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:55:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:55:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:55:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:55:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:55:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-15T00:55:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-15T00:55:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:55:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:55:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:55:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:55:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:55:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:55:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:55:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:55:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:55:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:55:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:55:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:55:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-15T00:55:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-15T00:55:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:55:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:55:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:55:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:55:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:55:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:55:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:55:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:55:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:55:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:55:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:55:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:55:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:55:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:55:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:55:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:55:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:55:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:55:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:55:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:55:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:55:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:55:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:55:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:55:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:55:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-09-15T00:55:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:55:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:55:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:55:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:55:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:55:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:55:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:55:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-15T00:55:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:55:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:55:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:55:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-09-15T00:55:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:55:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-15T00:55:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_2.status 2024-09-15T00:55:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:55:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_2.status 2024-09-15T00:55:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-15T00:55:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_2.status 2024-09-15T00:55:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:55:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.status 2024-09-15T00:55:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-15T00:55:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-15T00:55:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 88 2024-09-15T00:55:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_2.status 2024-09-15T00:55:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:55:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_2.status 2024-09-15T00:55:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-15T00:55:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:55:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:55:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=88 2024-09-15T00:55:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:55:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:55:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:55:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.status 2024-09-15T00:55:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.status 2024-09-15T00:55:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-15T00:55:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:55:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:55:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:55:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:55:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:55:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:55:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:55:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 88 = '' 2024-09-15T00:55:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:55:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:55:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.status 2024-09-15T00:55:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.status 2024-09-15T00:55:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-15T00:55:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:55:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:55:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:55:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:55:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:55:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 88 = '' 2024-09-15T00:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:55:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.status 2024-09-15T00:55:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.status 2024-09-15T00:55:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-15T00:55:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:55:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:55:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=88 2024-09-15T00:55:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 88 = 88 2024-09-15T00:55:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:55:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-15T00:55:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:55:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:55:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:55:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:55:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:55:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:55:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-15T00:56:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_2': 2024-09-15T00:56:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 11828a0d18b5 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.11828a0d18b5 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:46:53 2024 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:46:53 2024 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:46:53 2024 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:56:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:56:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 52f7a6a8-1ded-4b8e-b848-e1f2de6a62b1 2024-09-15T00:56:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:56:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-15T00:56:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T00:56:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:56:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_2.export 2024-09-15T00:56:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.export 2024-09-15T00:56:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_2.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.export 2024-09-15T00:56:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_2.export 2024-09-15T00:56:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.export 2024-09-15T00:56:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_2.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.export 2024-09-15T00:56:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_2.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_2.export 2024-09-15T00:56:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:56:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_3 2024-09-15T00:56:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-15T00:56:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-15T00:56:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:56:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:56:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:56:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-15T00:56:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:56:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:56:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:56:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:56:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:56:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:56:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:56:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:56:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-15T00:56:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:56:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:56:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:56:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:56:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:56:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:56:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:56:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:56:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-15T00:56:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-15T00:56:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:56:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:56:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:56:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-09-15T00:56:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-15T00:56:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:56:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:56:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:56:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2024-09-15T00:56:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:56:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:56:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:56:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:56:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-09-15T00:56:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-15T00:56:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_3.status 2024-09-15T00:56:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_3.status 2024-09-15T00:56:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-15T00:56:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_3.status 2024-09-15T00:56:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.status 2024-09-15T00:56:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-15T00:56:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-15T00:56:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 90 2024-09-15T00:56:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_3.status 2024-09-15T00:56:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_3.status 2024-09-15T00:56:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-15T00:56:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:56:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=90 2024-09-15T00:56:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:56:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.status 2024-09-15T00:56:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.status 2024-09-15T00:56:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-15T00:56:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=37 2024-09-15T00:56:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 90 = 37 2024-09-15T00:56:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:56:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.status 2024-09-15T00:56:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.status 2024-09-15T00:56:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-15T00:56:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 90 = '' 2024-09-15T00:56:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:56:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.status 2024-09-15T00:56:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.status 2024-09-15T00:56:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-15T00:56:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 90 = '' 2024-09-15T00:56:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:56:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.status 2024-09-15T00:56:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.status 2024-09-15T00:56:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-15T00:56:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=90 2024-09-15T00:56:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 90 = 90 2024-09-15T00:56:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-15T00:56:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:56:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_3': 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1182d1e344be 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1182d1e344be 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:56:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:47:05 2024 2024-09-15T00:56:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:47:05 2024 2024-09-15T00:56:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:47:05 2024 2024-09-15T00:56:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:56:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:56:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:56:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 3262a16a-009b-44a2-b91f-62ecc486851b 2024-09-15T00:56:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:56:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-15T00:56:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T00:56:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:56:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_3.export 2024-09-15T00:56:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.export 2024-09-15T00:56:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_3.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.export 2024-09-15T00:56:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_3.export 2024-09-15T00:56:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.export 2024-09-15T00:56:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_3.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.export 2024-09-15T00:56:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_3.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_3.export 2024-09-15T00:56:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:56:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_4 2024-09-15T00:56:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-15T00:56:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:56:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-15T00:56:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-15T00:56:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:56:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-15T00:56:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:56:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:56:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:56:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-15T00:56:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:56:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:56:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:56:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:56:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:56:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:56:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:56:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:56:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-15T00:56:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:56:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:56:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:56:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:56:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-15T00:56:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-09-15T00:56:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:56:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:56:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:56:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-15T00:56:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-09-15T00:56:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:56:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:56:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:56:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-09-15T00:56:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:56:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:56:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:56:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:56:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:56:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:56:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-15T00:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:56:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:56:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-09-15T00:56:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-15T00:56:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_4.status 2024-09-15T00:56:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_4.status 2024-09-15T00:56:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-15T00:56:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_4.status 2024-09-15T00:56:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.status 2024-09-15T00:56:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-15T00:56:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-15T00:56:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 92 2024-09-15T00:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_4.status 2024-09-15T00:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_4.status 2024-09-15T00:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-15T00:56:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:56:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=92 2024-09-15T00:56:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:56:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.status 2024-09-15T00:56:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.status 2024-09-15T00:56:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-15T00:56:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=38 2024-09-15T00:56:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 92 = 38 2024-09-15T00:56:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:56:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.status 2024-09-15T00:56:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.status 2024-09-15T00:56:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-15T00:56:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 92 = '' 2024-09-15T00:56:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:56:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.status 2024-09-15T00:56:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.status 2024-09-15T00:56:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-15T00:56:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 92 = '' 2024-09-15T00:56:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:56:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.status 2024-09-15T00:56:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.status 2024-09-15T00:56:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-15T00:56:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=92 2024-09-15T00:56:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 92 = 92 2024-09-15T00:56:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-15T00:56:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:56:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-15T00:56:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_4': 2024-09-15T00:56:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:56:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:56:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:56:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 11822ec4e7e1 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.11822ec4e7e1 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:47:13 2024 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:47:13 2024 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:47:13 2024 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 51352989-1223-414e-86a9-443c3c234183 2024-09-15T00:56:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:56:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-15T00:56:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T00:56:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:56:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_4.export 2024-09-15T00:56:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.export 2024-09-15T00:56:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_4.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.export 2024-09-15T00:56:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_4.export 2024-09-15T00:56:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.export 2024-09-15T00:56:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_4.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.export 2024-09-15T00:56:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_4.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_4.export 2024-09-15T00:56:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:56:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_5 2024-09-15T00:56:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-15T00:56:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:56:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-15T00:56:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-15T00:56:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:56:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-15T00:56:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:56:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:56:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:56:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-15T00:56:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:56:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:56:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:56:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:56:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:56:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:56:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:56:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:56:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:56:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:56:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:56:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-15T00:56:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-15T00:56:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:56:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:56:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:56:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-15T00:56:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-15T00:56:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-15T00:56:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:56:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:56:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:56:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:56:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-15T00:56:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:56:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:56:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:56:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:56:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:56:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:56:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-15T00:56:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_5.status 2024-09-15T00:56:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_5.status 2024-09-15T00:56:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-15T00:56:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_5.status 2024-09-15T00:56:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.status 2024-09-15T00:56:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-15T00:56:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-15T00:56:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 94 2024-09-15T00:56:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_5.status 2024-09-15T00:56:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_5.status 2024-09-15T00:56:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-15T00:56:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:56:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=94 2024-09-15T00:56:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:56:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.status 2024-09-15T00:56:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.status 2024-09-15T00:56:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-15T00:56:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=39 2024-09-15T00:56:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 94 = 39 2024-09-15T00:56:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:56:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.status 2024-09-15T00:56:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.status 2024-09-15T00:56:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-15T00:56:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 94 = '' 2024-09-15T00:56:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:56:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.status 2024-09-15T00:56:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.status 2024-09-15T00:56:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-15T00:56:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 94 = '' 2024-09-15T00:56:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:56:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.status 2024-09-15T00:56:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.status 2024-09-15T00:56:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-15T00:56:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=94 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 94 = 94 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-15T00:56:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_5': 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 11823ea6af06 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.11823ea6af06 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:47:28 2024 2024-09-15T00:56:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:47:28 2024 2024-09-15T00:56:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:47:28 2024 2024-09-15T00:56:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:56:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:56:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:56:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 8d62f872-96bf-4d6b-b93e-efad6f76511e 2024-09-15T00:56:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-15T00:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T00:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_5.export 2024-09-15T00:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.export 2024-09-15T00:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_5.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.export 2024-09-15T00:56:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_5.export 2024-09-15T00:56:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.export 2024-09-15T00:56:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_5.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.export 2024-09-15T00:56:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_5.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_5.export 2024-09-15T00:56:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:56:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_6 2024-09-15T00:56:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-15T00:56:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:56:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-15T00:56:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-15T00:56:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:56:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-15T00:56:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:56:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:56:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:56:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-15T00:56:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:56:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:56:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:56:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:56:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:56:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:56:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:56:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:56:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-15T00:56:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:56:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:56:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:56:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:56:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:56:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:56:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:56:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:56:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-15T00:56:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-15T00:56:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-15T00:56:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:56:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:56:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:56:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:56:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:56:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:56:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-15T00:56:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-15T00:56:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:56:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:56:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:56:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2024-09-15T00:56:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:56:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:56:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:56:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:56:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:56:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:56:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:56:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-15T00:56:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:56:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:56:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-09-15T00:56:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-15T00:56:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_6.status 2024-09-15T00:56:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_6.status 2024-09-15T00:56:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-15T00:56:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_6.status 2024-09-15T00:56:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.status 2024-09-15T00:56:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-15T00:56:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-15T00:56:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 96 2024-09-15T00:56:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_6.status 2024-09-15T00:56:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_6.status 2024-09-15T00:56:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-15T00:56:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:56:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=96 2024-09-15T00:56:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:56:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.status 2024-09-15T00:56:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.status 2024-09-15T00:56:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-15T00:56:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=40 2024-09-15T00:56:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 96 = 40 2024-09-15T00:56:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:56:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.status 2024-09-15T00:56:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.status 2024-09-15T00:56:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-15T00:56:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 96 = '' 2024-09-15T00:56:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:56:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.status 2024-09-15T00:56:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.status 2024-09-15T00:56:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-15T00:56:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 96 = '' 2024-09-15T00:56:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:56:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.status 2024-09-15T00:56:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.status 2024-09-15T00:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-15T00:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=96 2024-09-15T00:56:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 96 = 96 2024-09-15T00:56:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-15T00:56:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:56:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-15T00:56:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_6': 2024-09-15T00:56:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 118254a93b21 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.118254a93b21 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:47:42 2024 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:47:42 2024 2024-09-15T00:56:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:47:42 2024 2024-09-15T00:56:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:56:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:56:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:56:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 003fd3d0-584f-46ad-a08c-185239d8066b 2024-09-15T00:56:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:56:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-15T00:56:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T00:56:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:56:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_6.export 2024-09-15T00:56:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.export 2024-09-15T00:56:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_6.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.export 2024-09-15T00:56:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_6.export 2024-09-15T00:56:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.export 2024-09-15T00:56:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_6.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.export 2024-09-15T00:56:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_6.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_6.export 2024-09-15T00:56:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:56:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_7 2024-09-15T00:56:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-15T00:56:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-15T00:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:56:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-15T00:56:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:56:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:56:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:56:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:56:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:56:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:56:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:56:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:56:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-15T00:56:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:56:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:56:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:56:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:56:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:56:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:56:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:56:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:56:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-15T00:56:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-15T00:56:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:56:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-09-15T00:56:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-15T00:56:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:56:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:56:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:56:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-09-15T00:56:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:56:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:56:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:56:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:56:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:56:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:56:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:56:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-15T00:56:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_7.status 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_7.status 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_7.status 2024-09-15T00:56:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.status 2024-09-15T00:56:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-15T00:56:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-15T00:56:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 98 2024-09-15T00:56:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_7.status 2024-09-15T00:56:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_7.status 2024-09-15T00:56:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-15T00:56:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:56:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=98 2024-09-15T00:56:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:56:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.status 2024-09-15T00:56:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.status 2024-09-15T00:56:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-15T00:56:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=41 2024-09-15T00:56:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 98 = 41 2024-09-15T00:56:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:56:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.status 2024-09-15T00:56:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.status 2024-09-15T00:56:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-15T00:56:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 98 = '' 2024-09-15T00:56:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:56:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.status 2024-09-15T00:56:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.status 2024-09-15T00:56:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-15T00:56:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 98 = '' 2024-09-15T00:56:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:56:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.status 2024-09-15T00:56:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.status 2024-09-15T00:56:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-15T00:56:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=98 2024-09-15T00:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 98 = 98 2024-09-15T00:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-15T00:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:56:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-15T00:56:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_7': 2024-09-15T00:56:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:56:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:56:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:56:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 118243d4dfe9 2024-09-15T00:56:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.118243d4dfe9 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:47:48 2024 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:47:48 2024 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:47:48 2024 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 35b7b6ec-da56-4f63-91b0-35afde53e670 2024-09-15T00:56:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:56:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-15T00:56:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T00:56:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:56:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_7.export 2024-09-15T00:56:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.export 2024-09-15T00:56:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_7.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.export 2024-09-15T00:56:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_7.export 2024-09-15T00:56:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.export 2024-09-15T00:56:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_7.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.export 2024-09-15T00:56:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_7.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_7.export 2024-09-15T00:56:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:56:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_8 2024-09-15T00:56:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-15T00:56:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-15T00:56:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:56:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:56:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:56:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-15T00:56:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:56:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:56:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:56:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:56:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:56:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:56:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:56:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:56:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-15T00:56:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:56:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:56:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:56:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:56:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:56:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:56:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:56:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:56:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-15T00:56:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-15T00:56:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-15T00:56:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:56:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:56:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:56:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:56:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:56:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:56:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:56:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-15T00:56:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-15T00:56:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2024-09-15T00:56:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:56:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:56:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:56:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:56:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:56:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:56:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:56:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2024-09-15T00:56:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:56:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:56:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:56:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:56:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:56:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-15T00:56:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_8.status 2024-09-15T00:56:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_8.status 2024-09-15T00:56:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-15T00:56:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_8.status 2024-09-15T00:56:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.status 2024-09-15T00:56:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-15T00:56:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-15T00:56:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 100 2024-09-15T00:56:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_8.status 2024-09-15T00:56:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_8.status 2024-09-15T00:56:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-15T00:56:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:56:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=100 2024-09-15T00:56:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:56:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.status 2024-09-15T00:56:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.status 2024-09-15T00:56:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-15T00:56:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=42 2024-09-15T00:56:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 100 = 42 2024-09-15T00:56:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:56:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.status 2024-09-15T00:56:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.status 2024-09-15T00:56:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-15T00:56:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 100 = '' 2024-09-15T00:56:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:56:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.status 2024-09-15T00:56:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.status 2024-09-15T00:56:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-15T00:56:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 100 = '' 2024-09-15T00:56:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:56:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.status 2024-09-15T00:56:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.status 2024-09-15T00:56:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-15T00:56:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=100 2024-09-15T00:56:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 100 = 100 2024-09-15T00:56:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-15T00:56:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:56:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-15T00:56:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_8': 2024-09-15T00:56:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:56:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:56:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:56:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 118232a55df9 2024-09-15T00:56:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.118232a55df9 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:03 2024 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:03 2024 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:03 2024 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 5728982c-d76d-410e-bc93-a9b0b0701e25 2024-09-15T00:56:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:56:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-15T00:56:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T00:56:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:56:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_8.export 2024-09-15T00:56:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.export 2024-09-15T00:56:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_8.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.export 2024-09-15T00:56:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_8.export 2024-09-15T00:56:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.export 2024-09-15T00:56:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_8.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.export 2024-09-15T00:56:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_8.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_8.export 2024-09-15T00:56:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:56:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_9 2024-09-15T00:56:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-15T00:56:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:56:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-15T00:56:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-15T00:56:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:56:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-15T00:56:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:56:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:56:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:56:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-15T00:56:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:56:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:56:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:56:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:56:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:56:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:56:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:56:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:56:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-15T00:56:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:56:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:56:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:56:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:56:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-15T00:56:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-15T00:56:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:56:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:56:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:56:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-09-15T00:56:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-15T00:56:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2024-09-15T00:56:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:56:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:56:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:56:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:56:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2024-09-15T00:56:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:56:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:56:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:56:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:56:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:56:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:56:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:56:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-15T00:56:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_9.status 2024-09-15T00:56:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_9.status 2024-09-15T00:56:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-15T00:56:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_9.status 2024-09-15T00:56:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:56:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.status 2024-09-15T00:56:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-15T00:56:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-15T00:56:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 102 2024-09-15T00:56:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_9.status 2024-09-15T00:56:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_9.status 2024-09-15T00:56:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-15T00:56:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:56:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=102 2024-09-15T00:56:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:56:52.850 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4507377352537525) > weight (0.25) 2024-09-15T00:56:52.851 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-15T00:56:52.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-15T00:56:52.851 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-15T00:56:52.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T00:56:52.850+0000 7f3b47f30640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 48824) UID: 1000 2024-09-15T00:56:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.status 2024-09-15T00:56:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.status 2024-09-15T00:56:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-15T00:56:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=43 2024-09-15T00:56:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 102 = 43 2024-09-15T00:56:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:56:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.status 2024-09-15T00:56:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.status 2024-09-15T00:56:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-15T00:56:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 102 = '' 2024-09-15T00:56:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:56:53.852 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-15T00:56:53.853 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:56:53.853 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-15T00:56:53.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-15T00:56:53.854 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-15T00:56:53.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-15T00:56:53.857 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-15T00:56:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.status 2024-09-15T00:56:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.status 2024-09-15T00:56:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-15T00:56:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:56:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:56:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:56:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:56:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:56:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 102 = '' 2024-09-15T00:56:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:56:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:56:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.status 2024-09-15T00:56:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.status 2024-09-15T00:56:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-15T00:56:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:56:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:56:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=102 2024-09-15T00:56:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 102 = 102 2024-09-15T00:56:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-15T00:56:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:56:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-15T00:56:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_9': 2024-09-15T00:56:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:56:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:56:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:56:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 11827fb7ac85 2024-09-15T00:56:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.11827fb7ac85 2024-09-15T00:56:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:11 2024 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:11 2024 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:11 2024 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 344e6c94-9331-4900-a47f-b1036d651c77 2024-09-15T00:56:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:56:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-15T00:56:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T00:56:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:56:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_9.export 2024-09-15T00:56:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.export 2024-09-15T00:56:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_9.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.export 2024-09-15T00:56:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_9.export 2024-09-15T00:56:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.export 2024-09-15T00:56:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:56:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_9.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.export 2024-09-15T00:56:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_9.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_9.export 2024-09-15T00:56:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:56:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:56:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_10 2024-09-15T00:56:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-15T00:56:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:56:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:56:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:56:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:56:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-15T00:56:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:56:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-15T00:56:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:56:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:56:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:56:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:56:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-15T00:56:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:56:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:56:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:56:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:56:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:56:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:56:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-15T00:56:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:56:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:56:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:56:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:56:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:56:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:56:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:56:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:56:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:56:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-15T00:56:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:56:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:56:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:56:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:56:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:56:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:56:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:56:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:56:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:56:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:56:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:56:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-15T00:56:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-15T00:56:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:56:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:56:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:56:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:56:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:56:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:56:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:56:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:56:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:56:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:56:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:56:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:56:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-15T00:57:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-15T00:57:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-09-15T00:57:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:57:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:57:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:57:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:57:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-09-15T00:57:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:57:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:57:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:57:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:57:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:57:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-15T00:57:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_10.status 2024-09-15T00:57:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_10.status 2024-09-15T00:57:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-15T00:57:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_10.status 2024-09-15T00:57:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.status 2024-09-15T00:57:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-15T00:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-15T00:57:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 104 2024-09-15T00:57:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_10.status 2024-09-15T00:57:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:57:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_10.status 2024-09-15T00:57:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-15T00:57:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:57:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=104 2024-09-15T00:57:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:57:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.status 2024-09-15T00:57:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:57:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.status 2024-09-15T00:57:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-15T00:57:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 104 = '' 2024-09-15T00:57:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:57:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.status 2024-09-15T00:57:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:57:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.status 2024-09-15T00:57:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-15T00:57:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 104 = '' 2024-09-15T00:57:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:57:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.status 2024-09-15T00:57:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:57:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.status 2024-09-15T00:57:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-15T00:57:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=104 2024-09-15T00:57:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 104 = 104 2024-09-15T00:57:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-15T00:57:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:57:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:57:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-15T00:57:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_10': 2024-09-15T00:57:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1182a7332b74 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1182a7332b74 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:20 2024 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:20 2024 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:20 2024 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:57:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:57:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 7902b9a5-d221-4297-9894-a93a67ef6ff7 2024-09-15T00:57:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-15T00:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T00:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_10.export 2024-09-15T00:57:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.export 2024-09-15T00:57:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_10.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.export 2024-09-15T00:57:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_10.export 2024-09-15T00:57:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.export 2024-09-15T00:57:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_10.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.export 2024-09-15T00:57:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_10.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_10.export 2024-09-15T00:57:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_11 2024-09-15T00:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-15T00:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:57:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-15T00:57:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:57:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:57:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:57:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-15T00:57:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:57:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:57:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:57:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:57:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:57:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:57:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:57:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:57:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:57:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:57:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:57:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-15T00:57:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-15T00:57:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-09-15T00:57:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:57:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:57:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:57:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:57:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:57:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:57:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:57:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-09-15T00:57:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-15T00:57:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:57:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:57:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:57:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-09-15T00:57:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:57:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:57:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:57:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:57:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:57:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-15T00:57:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_11.status 2024-09-15T00:57:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_11.status 2024-09-15T00:57:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-15T00:57:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_11.status 2024-09-15T00:57:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.status 2024-09-15T00:57:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-15T00:57:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-15T00:57:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 106 2024-09-15T00:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_11.status 2024-09-15T00:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_11.status 2024-09-15T00:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-15T00:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:57:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=106 2024-09-15T00:57:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:57:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.status 2024-09-15T00:57:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.status 2024-09-15T00:57:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-15T00:57:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 106 = '' 2024-09-15T00:57:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:57:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.status 2024-09-15T00:57:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.status 2024-09-15T00:57:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-15T00:57:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 106 = '' 2024-09-15T00:57:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:57:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.status 2024-09-15T00:57:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.status 2024-09-15T00:57:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-15T00:57:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=106 2024-09-15T00:57:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 106 = 106 2024-09-15T00:57:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-15T00:57:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:57:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:10.060 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6498570728595571) > weight (0.25) 2024-09-15T00:57:10.060 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6563016210567193) > weight (0.25) 2024-09-15T00:57:10.060 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9519201141146616) > weight (0.25) 2024-09-15T00:57:10.060 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5437400552330247) > weight (0.25) 2024-09-15T00:57:10.061 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-09-15T00:57:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-15T00:57:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_11': 2024-09-15T00:57:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126ec8b11a5c 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126ec8b11a5c 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:31 2024 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:31 2024 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:31 2024 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:57:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:57:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 4d7f7b50-55a3-40e9-8e7e-87fb1c8cf94e 2024-09-15T00:57:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:57:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-15T00:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T00:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_11.export 2024-09-15T00:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.export 2024-09-15T00:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_11.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.export 2024-09-15T00:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_11.export 2024-09-15T00:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.export 2024-09-15T00:57:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_11.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.export 2024-09-15T00:57:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_11.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_11.export 2024-09-15T00:57:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:57:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_12 2024-09-15T00:57:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-15T00:57:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:57:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-15T00:57:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-15T00:57:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:57:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-15T00:57:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:57:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:57:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:57:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-15T00:57:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:57:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:57:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:57:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:57:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:57:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:57:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:57:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:57:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:57:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-15T00:57:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-15T00:57:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-15T00:57:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:57:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:57:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:57:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-15T00:57:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-15T00:57:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:57:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:57:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:57:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2024-09-15T00:57:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:57:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:57:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:57:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:57:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:57:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-09-15T00:57:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-15T00:57:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_12.status 2024-09-15T00:57:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_12.status 2024-09-15T00:57:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-15T00:57:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_12.status 2024-09-15T00:57:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.status 2024-09-15T00:57:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-15T00:57:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-15T00:57:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 108 2024-09-15T00:57:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_12.status 2024-09-15T00:57:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_12.status 2024-09-15T00:57:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-15T00:57:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:57:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=108 2024-09-15T00:57:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:57:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.status 2024-09-15T00:57:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.status 2024-09-15T00:57:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-15T00:57:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 108 = '' 2024-09-15T00:57:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:57:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.status 2024-09-15T00:57:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.status 2024-09-15T00:57:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-15T00:57:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 108 = '' 2024-09-15T00:57:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.status 2024-09-15T00:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.status 2024-09-15T00:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-15T00:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=108 2024-09-15T00:57:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 108 = 108 2024-09-15T00:57:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-15T00:57:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:57:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-15T00:57:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_12': 2024-09-15T00:57:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:57:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e53e78d49 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e53e78d49 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:38 2024 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:38 2024 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:38 2024 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: a1914ce5-652f-465e-9c3f-e18ac5ac2ca8 2024-09-15T00:57:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:57:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-15T00:57:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T00:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_12.export 2024-09-15T00:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.export 2024-09-15T00:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_12.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.export 2024-09-15T00:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_12.export 2024-09-15T00:57:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.export 2024-09-15T00:57:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_12.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.export 2024-09-15T00:57:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_12.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_12.export 2024-09-15T00:57:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:57:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_13 2024-09-15T00:57:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-15T00:57:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:57:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-15T00:57:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-15T00:57:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:57:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:20.162 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.160+0000 7fc49cacc640 -1 reset not still connected to 0x560c53e39040 2024-09-15T00:57:20.163 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.160+0000 7fc49cacc640 -1 reset not still connected to 0x560c573aa410 2024-09-15T00:57:20.163 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.160+0000 7fc49cacc640 -1 reset not still connected to 0x560c58b9c1a0 2024-09-15T00:57:20.163 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.160+0000 7fc49cacc640 -1 reset not still connected to 0x560c59eaed00 2024-09-15T00:57:20.163 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.160+0000 7fc49cacc640 -1 reset not still connected to 0x560c5a88a410 2024-09-15T00:57:20.163 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.160+0000 7fc49cacc640 -1 reset not still connected to 0x560c5c381860 2024-09-15T00:57:20.163 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.160+0000 7fc49cacc640 -1 reset not still connected to 0x560c5d7e7110 2024-09-15T00:57:20.167 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.164+0000 7fc49cacc640 -1 reset not still connected to 0x560c5f7da410 2024-09-15T00:57:20.167 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.164+0000 7fc49cacc640 -1 reset not still connected to 0x560c5f905790 2024-09-15T00:57:20.167 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.164+0000 7fc49cacc640 -1 reset not still connected to 0x560c5fea91e0 2024-09-15T00:57:20.167 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.164+0000 7fc49cacc640 -1 reset not still connected to 0x560c611c9a00 2024-09-15T00:57:20.167 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.164+0000 7fc49cacc640 -1 reset not still connected to 0x560c62eb11e0 2024-09-15T00:57:20.167 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.164+0000 7fc49cacc640 -1 reset not still connected to 0x560c62ec95f0 2024-09-15T00:57:20.167 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.164+0000 7fc49cacc640 -1 reset not still connected to 0x560c66b06340 2024-09-15T00:57:20.170 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.168+0000 7fc49cacc640 -1 reset not still connected to 0x560c6bbd3380 2024-09-15T00:57:20.170 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T00:57:20.168+0000 7fc49cacc640 -1 reset not still connected to 0x560c6c7ced00 2024-09-15T00:57:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-15T00:57:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:57:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:57:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-15T00:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:57:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:57:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:57:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:57:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:57:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:57:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:57:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-15T00:57:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:57:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:57:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:57:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:57:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:57:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:57:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:57:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-15T00:57:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-15T00:57:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-09-15T00:57:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:57:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:57:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:57:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:57:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:57:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:57:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-09-15T00:57:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-15T00:57:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2024-09-15T00:57:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:57:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:57:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:57:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:57:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2024-09-15T00:57:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:57:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:57:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:57:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:57:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:57:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-15T00:57:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_13.status 2024-09-15T00:57:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_13.status 2024-09-15T00:57:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-15T00:57:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_13.status 2024-09-15T00:57:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.status 2024-09-15T00:57:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-15T00:57:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-15T00:57:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 110 2024-09-15T00:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_13.status 2024-09-15T00:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_13.status 2024-09-15T00:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-15T00:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:57:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=110 2024-09-15T00:57:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:57:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.status 2024-09-15T00:57:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.status 2024-09-15T00:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-15T00:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 110 = '' 2024-09-15T00:57:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:57:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.status 2024-09-15T00:57:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.status 2024-09-15T00:57:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-15T00:57:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 110 = '' 2024-09-15T00:57:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:57:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.status 2024-09-15T00:57:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.status 2024-09-15T00:57:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-15T00:57:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=110 2024-09-15T00:57:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 110 = 110 2024-09-15T00:57:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-15T00:57:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:57:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-15T00:57:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_13': 2024-09-15T00:57:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:57:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:57:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126ee7f6c61b 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126ee7f6c61b 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:46 2024 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:46 2024 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:46 2024 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:57:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 66207760-d8d4-44f5-80a1-fdab006bbc16 2024-09-15T00:57:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:57:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-15T00:57:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T00:57:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:57:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_13.export 2024-09-15T00:57:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.export 2024-09-15T00:57:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_13.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.export 2024-09-15T00:57:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_13.export 2024-09-15T00:57:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.export 2024-09-15T00:57:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_13.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.export 2024-09-15T00:57:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_13.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_13.export 2024-09-15T00:57:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:57:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_14 2024-09-15T00:57:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-15T00:57:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:57:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-15T00:57:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:57:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:57:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:57:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:57:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-15T00:57:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:57:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:57:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:57:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:57:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:57:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:57:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:57:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:57:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-15T00:57:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-15T00:57:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-15T00:57:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:57:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:57:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:57:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:57:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-15T00:57:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-15T00:57:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:57:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-09-15T00:57:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:57:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:57:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:57:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:57:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:57:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-15T00:57:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_14.status 2024-09-15T00:57:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_14.status 2024-09-15T00:57:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-15T00:57:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_14.status 2024-09-15T00:57:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.status 2024-09-15T00:57:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-15T00:57:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-15T00:57:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 112 2024-09-15T00:57:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_14.status 2024-09-15T00:57:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_14.status 2024-09-15T00:57:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-15T00:57:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:57:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=112 2024-09-15T00:57:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:57:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.status 2024-09-15T00:57:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.status 2024-09-15T00:57:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-15T00:57:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 112 = '' 2024-09-15T00:57:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:57:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.status 2024-09-15T00:57:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.status 2024-09-15T00:57:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-15T00:57:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 112 = '' 2024-09-15T00:57:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:57:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.status 2024-09-15T00:57:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.status 2024-09-15T00:57:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-15T00:57:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=112 2024-09-15T00:57:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 112 = 112 2024-09-15T00:57:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-15T00:57:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:57:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-15T00:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_14': 2024-09-15T00:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e58be1233 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e58be1233 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:48:59 2024 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:48:59 2024 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:48:59 2024 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 5a261e22-728a-47f5-ad1c-cec823ae7b9a 2024-09-15T00:57:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:57:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-15T00:57:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T00:57:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:57:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_14.export 2024-09-15T00:57:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.export 2024-09-15T00:57:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_14.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.export 2024-09-15T00:57:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_14.export 2024-09-15T00:57:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.export 2024-09-15T00:57:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_14.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.export 2024-09-15T00:57:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_14.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_14.export 2024-09-15T00:57:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:57:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_15 2024-09-15T00:57:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-15T00:57:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:57:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-15T00:57:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-15T00:57:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:57:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-15T00:57:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:57:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:57:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:57:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-15T00:57:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:57:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:57:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:57:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:57:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:57:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:57:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:57:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:57:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-15T00:57:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:57:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:57:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:57:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:57:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:57:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:57:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-15T00:57:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-15T00:57:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-09-15T00:57:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:57:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:57:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:57:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:57:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-09-15T00:57:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-15T00:57:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2024-09-15T00:57:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:57:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:57:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:57:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:57:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:57:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:57:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:57:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:57:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2024-09-15T00:57:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:57:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:57:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:57:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:57:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:57:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:57:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:57:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-15T00:57:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_15.status 2024-09-15T00:57:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_15.status 2024-09-15T00:57:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-15T00:57:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_15.status 2024-09-15T00:57:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.status 2024-09-15T00:57:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-15T00:57:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-15T00:57:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 114 2024-09-15T00:57:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_15.status 2024-09-15T00:57:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_15.status 2024-09-15T00:57:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-15T00:57:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:57:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=114 2024-09-15T00:57:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:57:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.status 2024-09-15T00:57:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.status 2024-09-15T00:57:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-15T00:57:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 114 = '' 2024-09-15T00:57:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:57:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.status 2024-09-15T00:57:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.status 2024-09-15T00:57:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-15T00:57:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 114 = '' 2024-09-15T00:57:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:57:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.status 2024-09-15T00:57:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.status 2024-09-15T00:57:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-15T00:57:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=114 2024-09-15T00:57:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 114 = 114 2024-09-15T00:57:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-15T00:57:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:57:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_15': 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126eaa915476 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126eaa915476 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:49:06 2024 2024-09-15T00:57:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:49:06 2024 2024-09-15T00:57:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:49:06 2024 2024-09-15T00:57:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:57:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:57:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:57:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 0df0e3a0-f71b-4bbc-8631-00ed1ea29cb7 2024-09-15T00:57:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:57:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-15T00:57:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T00:57:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:57:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_15.export 2024-09-15T00:57:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.export 2024-09-15T00:57:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_15.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.export 2024-09-15T00:57:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_15.export 2024-09-15T00:57:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.export 2024-09-15T00:57:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_15.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.export 2024-09-15T00:57:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_15.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_15.export 2024-09-15T00:57:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:57:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_16 2024-09-15T00:57:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-15T00:57:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:57:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-15T00:57:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-15T00:57:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:57:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-15T00:57:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:57:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:57:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:57:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-15T00:57:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:57:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:57:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:57:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:57:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:57:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:57:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:57:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:57:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-15T00:57:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:57:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:57:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:57:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:57:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:57:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:57:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:57:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:57:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-15T00:57:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-15T00:57:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-15T00:57:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:57:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:57:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:57:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:57:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-15T00:57:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-15T00:57:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:57:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:57:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:57:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-09-15T00:57:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:57:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:57:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:57:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:57:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:57:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-15T00:57:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_16.status 2024-09-15T00:57:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_16.status 2024-09-15T00:57:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-15T00:57:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_16.status 2024-09-15T00:57:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.status 2024-09-15T00:57:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-15T00:57:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-15T00:57:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 116 2024-09-15T00:57:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_16.status 2024-09-15T00:57:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_16.status 2024-09-15T00:57:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-15T00:57:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:57:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=116 2024-09-15T00:57:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:57:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.status 2024-09-15T00:57:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.status 2024-09-15T00:57:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-15T00:57:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=50 2024-09-15T00:57:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 116 = 50 2024-09-15T00:57:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:57:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.status 2024-09-15T00:57:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.status 2024-09-15T00:57:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-15T00:57:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 116 = '' 2024-09-15T00:57:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:57:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.status 2024-09-15T00:57:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.status 2024-09-15T00:57:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-15T00:57:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 116 = '' 2024-09-15T00:57:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:57:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.status 2024-09-15T00:57:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.status 2024-09-15T00:57:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-15T00:57:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=116 2024-09-15T00:57:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 116 = 116 2024-09-15T00:57:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-15T00:57:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:57:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-15T00:57:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_16': 2024-09-15T00:57:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:57:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:57:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:57:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126ea5dfc8f 2024-09-15T00:57:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126ea5dfc8f 2024-09-15T00:57:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:57:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:57:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:57:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:57:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:49:16 2024 2024-09-15T00:57:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:49:16 2024 2024-09-15T00:57:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:49:16 2024 2024-09-15T00:57:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:57:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:57:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:57:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: b1a838da-8633-4b6c-a717-7c58db5d3618 2024-09-15T00:57:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:57:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-15T00:57:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T00:57:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:57:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_16.export 2024-09-15T00:57:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.export 2024-09-15T00:57:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_16.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.export 2024-09-15T00:57:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_16.export 2024-09-15T00:57:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.export 2024-09-15T00:57:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_16.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.export 2024-09-15T00:57:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_16.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_16.export 2024-09-15T00:57:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:57:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_17 2024-09-15T00:57:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-15T00:57:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:57:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-15T00:57:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-15T00:57:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:57:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-15T00:57:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:57:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:57:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:57:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-15T00:57:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:57:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:57:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:57:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:57:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:57:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:57:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:57:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:57:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:57:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-15T00:57:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-15T00:57:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-09-15T00:57:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:57:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:57:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:57:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:57:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:57:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:57:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:57:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:57:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:57:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:57:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:57:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:57:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-09-15T00:57:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-15T00:57:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2024-09-15T00:57:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:57:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:57:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2024-09-15T00:57:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:57:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:57:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:57:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:57:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:57:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:57:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-15T00:57:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_17.status 2024-09-15T00:57:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_17.status 2024-09-15T00:57:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-15T00:57:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_17.status 2024-09-15T00:57:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.status 2024-09-15T00:57:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-15T00:57:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-15T00:57:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 118 2024-09-15T00:57:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_17.status 2024-09-15T00:57:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_17.status 2024-09-15T00:57:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-15T00:57:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:57:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=118 2024-09-15T00:57:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:57:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.status 2024-09-15T00:57:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.status 2024-09-15T00:57:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-15T00:57:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=51 2024-09-15T00:57:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 118 = 51 2024-09-15T00:57:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:57:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.status 2024-09-15T00:57:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.status 2024-09-15T00:57:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-15T00:57:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 118 = '' 2024-09-15T00:57:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:57:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.status 2024-09-15T00:57:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.status 2024-09-15T00:57:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-15T00:57:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 118 = '' 2024-09-15T00:57:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:57:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.status 2024-09-15T00:57:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.status 2024-09-15T00:57:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-15T00:57:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=118 2024-09-15T00:57:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 118 = 118 2024-09-15T00:57:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-15T00:57:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:57:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-15T00:57:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_17': 2024-09-15T00:57:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:57:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:57:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:57:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e67bc6f43 2024-09-15T00:57:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e67bc6f43 2024-09-15T00:57:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:57:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:57:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:57:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:57:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:49:29 2024 2024-09-15T00:57:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:49:29 2024 2024-09-15T00:57:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:49:29 2024 2024-09-15T00:57:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:57:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:57:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:57:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 9a3b2ff6-9972-4b36-a5a7-25c8321572a0 2024-09-15T00:57:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:57:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-15T00:57:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T00:57:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:57:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_17.export 2024-09-15T00:57:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.export 2024-09-15T00:57:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_17.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.export 2024-09-15T00:57:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_17.export 2024-09-15T00:57:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.export 2024-09-15T00:57:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:57:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_17.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.export 2024-09-15T00:57:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_17.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_17.export 2024-09-15T00:57:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:57:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_18 2024-09-15T00:57:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-15T00:57:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:57:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:57:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:57:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-15T00:57:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:57:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-15T00:57:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:57:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:57:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:57:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:57:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-15T00:57:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:57:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:57:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:57:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:57:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-15T00:57:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:57:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:57:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:57:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:57:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:57:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:57:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:57:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:57:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:57:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:57:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:57:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:57:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:57:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:57:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:57:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-15T00:57:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-15T00:57:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-15T00:57:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:57:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:57:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:57:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:57:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:57:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:57:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:57:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:57:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:57:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:57:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:57:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-15T00:57:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-15T00:57:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:57:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:57:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:57:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:57:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:57:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:57:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:57:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-09-15T00:57:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:57:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:57:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:57:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:57:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:57:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:57:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:57:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:57:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-15T00:57:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:57:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:57:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:57:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-09-15T00:57:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:57:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:57:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-15T00:57:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_18.status 2024-09-15T00:57:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_18.status 2024-09-15T00:57:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-15T00:57:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_18.status 2024-09-15T00:57:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:57:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.status 2024-09-15T00:57:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-15T00:57:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:57:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-15T00:57:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 120 2024-09-15T00:57:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_18.status 2024-09-15T00:57:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:57:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:57:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_18.status 2024-09-15T00:57:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-15T00:57:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:57:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=120 2024-09-15T00:57:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:57:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.status 2024-09-15T00:57:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:57:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.status 2024-09-15T00:57:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-15T00:57:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=52 2024-09-15T00:57:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 120 = 52 2024-09-15T00:57:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:57:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.status 2024-09-15T00:57:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:57:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:57:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:57:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.status 2024-09-15T00:57:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-15T00:57:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:57:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:57:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:57:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:57:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:57:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:57:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:57:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 120 = '' 2024-09-15T00:57:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:57:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:58:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.status 2024-09-15T00:58:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:58:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.status 2024-09-15T00:58:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-15T00:58:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=120 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 120 = 120 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-15T00:58:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_18': 2024-09-15T00:58:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:58:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e92111d 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e92111d 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:49:40 2024 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:49:40 2024 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:49:40 2024 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:58:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:58:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 8c63bcd3-d5d8-46ca-9685-256e72243858 2024-09-15T00:58:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:58:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-15T00:58:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T00:58:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:58:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_18.export 2024-09-15T00:58:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.export 2024-09-15T00:58:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_18.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.export 2024-09-15T00:58:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_18.export 2024-09-15T00:58:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.export 2024-09-15T00:58:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_18.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.export 2024-09-15T00:58:01.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_18.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_18.export 2024-09-15T00:58:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:58:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_19 2024-09-15T00:58:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-15T00:58:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:58:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-15T00:58:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-15T00:58:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:58:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-15T00:58:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:58:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:58:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:58:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-15T00:58:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:58:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:58:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:58:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:58:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:58:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:58:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:58:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:58:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-15T00:58:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:58:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:58:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:58:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:58:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:58:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:58:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:58:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-15T00:58:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-09-15T00:58:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:58:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:58:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:58:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:58:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-09-15T00:58:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-09-15T00:58:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2024-09-15T00:58:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:58:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:58:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:58:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:58:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:58:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:58:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2024-09-15T00:58:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:58:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:58:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:58:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:58:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-15T00:58:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_19.status 2024-09-15T00:58:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_19.status 2024-09-15T00:58:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-15T00:58:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_19.status 2024-09-15T00:58:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.status 2024-09-15T00:58:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-15T00:58:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-15T00:58:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 122 2024-09-15T00:58:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_19.status 2024-09-15T00:58:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_19.status 2024-09-15T00:58:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-15T00:58:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:58:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=122 2024-09-15T00:58:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:58:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.status 2024-09-15T00:58:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.status 2024-09-15T00:58:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-15T00:58:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=53 2024-09-15T00:58:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 122 = 53 2024-09-15T00:58:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:58:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.status 2024-09-15T00:58:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.status 2024-09-15T00:58:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-15T00:58:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 122 = '' 2024-09-15T00:58:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:58:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.status 2024-09-15T00:58:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.status 2024-09-15T00:58:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-15T00:58:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 122 = '' 2024-09-15T00:58:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:58:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.status 2024-09-15T00:58:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.status 2024-09-15T00:58:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-15T00:58:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=122 2024-09-15T00:58:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 122 = 122 2024-09-15T00:58:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-15T00:58:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:58:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-15T00:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_19': 2024-09-15T00:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126ed172dd0d 2024-09-15T00:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126ed172dd0d 2024-09-15T00:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:58:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:49:52 2024 2024-09-15T00:58:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:49:52 2024 2024-09-15T00:58:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:49:52 2024 2024-09-15T00:58:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:58:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:58:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:58:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 237a420c-4d63-4519-a90b-522ac9d2b086 2024-09-15T00:58:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:58:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-15T00:58:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T00:58:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:58:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_19.export 2024-09-15T00:58:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.export 2024-09-15T00:58:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_19.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.export 2024-09-15T00:58:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_19.export 2024-09-15T00:58:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.export 2024-09-15T00:58:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_19.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.export 2024-09-15T00:58:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_19.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_19.export 2024-09-15T00:58:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:58:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_20 2024-09-15T00:58:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-15T00:58:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:58:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-15T00:58:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:58:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-15T00:58:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:58:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:58:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:58:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:58:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:58:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:58:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:58:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:58:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-15T00:58:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:58:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:58:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:58:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-15T00:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-15T00:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:58:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:58:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:58:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:58:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-15T00:58:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-15T00:58:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-09-15T00:58:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:58:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:58:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:58:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:58:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:58:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-09-15T00:58:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:58:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:58:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:58:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:58:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:58:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:58:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:58:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-15T00:58:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_20.status 2024-09-15T00:58:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_20.status 2024-09-15T00:58:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-15T00:58:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_20.status 2024-09-15T00:58:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.status 2024-09-15T00:58:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-15T00:58:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-15T00:58:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 124 2024-09-15T00:58:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_20.status 2024-09-15T00:58:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_20.status 2024-09-15T00:58:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-15T00:58:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:58:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=124 2024-09-15T00:58:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:58:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.status 2024-09-15T00:58:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.status 2024-09-15T00:58:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-15T00:58:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=54 2024-09-15T00:58:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 124 = 54 2024-09-15T00:58:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:58:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.status 2024-09-15T00:58:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.status 2024-09-15T00:58:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-15T00:58:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 124 = '' 2024-09-15T00:58:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:58:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.status 2024-09-15T00:58:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.status 2024-09-15T00:58:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-15T00:58:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 124 = '' 2024-09-15T00:58:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:58:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.status 2024-09-15T00:58:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.status 2024-09-15T00:58:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-15T00:58:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=124 2024-09-15T00:58:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 124 = 124 2024-09-15T00:58:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-15T00:58:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:58:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-15T00:58:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_20': 2024-09-15T00:58:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:58:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:58:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:58:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e8e3b12c7 2024-09-15T00:58:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e8e3b12c7 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:06 2024 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:06 2024 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:06 2024 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: ab51885a-03a7-4691-9f70-c8ead112d259 2024-09-15T00:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:58:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-15T00:58:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T00:58:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:58:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_20.export 2024-09-15T00:58:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.export 2024-09-15T00:58:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_20.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.export 2024-09-15T00:58:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_20.export 2024-09-15T00:58:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.export 2024-09-15T00:58:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_20.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.export 2024-09-15T00:58:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_20.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_20.export 2024-09-15T00:58:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:58:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_21 2024-09-15T00:58:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-15T00:58:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:58:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-15T00:58:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-15T00:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-15T00:58:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:58:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:58:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:58:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-15T00:58:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:58:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:58:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:58:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:58:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:58:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:58:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:58:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:58:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-15T00:58:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:58:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:58:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:58:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:58:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-15T00:58:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-15T00:58:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-09-15T00:58:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:58:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:58:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:58:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:58:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-09-15T00:58:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-15T00:58:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2024-09-15T00:58:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:58:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:58:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:58:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:58:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:58:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:58:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:58:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:58:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2024-09-15T00:58:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:58:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:58:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:58:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-09-15T00:58:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-15T00:58:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_21.status 2024-09-15T00:58:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_21.status 2024-09-15T00:58:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-15T00:58:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_21.status 2024-09-15T00:58:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.status 2024-09-15T00:58:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-15T00:58:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-15T00:58:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 126 2024-09-15T00:58:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_21.status 2024-09-15T00:58:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_21.status 2024-09-15T00:58:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-15T00:58:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:58:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=126 2024-09-15T00:58:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:58:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.status 2024-09-15T00:58:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.status 2024-09-15T00:58:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-15T00:58:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=55 2024-09-15T00:58:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 126 = 55 2024-09-15T00:58:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:58:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.status 2024-09-15T00:58:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.status 2024-09-15T00:58:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-15T00:58:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 126 = '' 2024-09-15T00:58:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:58:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.status 2024-09-15T00:58:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.status 2024-09-15T00:58:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-15T00:58:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 126 = '' 2024-09-15T00:58:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:58:24.858 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.338574100489087) > weight (0.25) 2024-09-15T00:58:24.858 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4447551458772039) > weight (0.25) 2024-09-15T00:58:24.858 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6059779258436292) > weight (0.25) 2024-09-15T00:58:24.858 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5421483671078111) > weight (0.25) 2024-09-15T00:58:24.858 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-15T00:58:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.status 2024-09-15T00:58:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.status 2024-09-15T00:58:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-15T00:58:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=126 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 126 = 126 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:58:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-15T00:58:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_21': 2024-09-15T00:58:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:58:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:58:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:58:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e70c8f279 2024-09-15T00:58:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e70c8f279 2024-09-15T00:58:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:58:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:58:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:58:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:58:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:16 2024 2024-09-15T00:58:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:16 2024 2024-09-15T00:58:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:16 2024 2024-09-15T00:58:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:58:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:58:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:58:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 69c7326f-a13b-465e-a7c0-8dec344e635a 2024-09-15T00:58:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:58:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-15T00:58:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T00:58:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:58:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_21.export 2024-09-15T00:58:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.export 2024-09-15T00:58:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_21.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.export 2024-09-15T00:58:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_21.export 2024-09-15T00:58:26.178 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.export 2024-09-15T00:58:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_21.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.export 2024-09-15T00:58:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_21.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_21.export 2024-09-15T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_22 2024-09-15T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-15T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-15T00:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-15T00:58:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:58:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-15T00:58:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:58:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:58:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:58:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-15T00:58:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:58:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:58:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:58:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:58:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:58:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:58:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:58:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:58:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:58:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:58:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:58:28.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:28.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:28.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:28.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-15T00:58:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-15T00:58:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:58:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:58:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-15T00:58:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-15T00:58:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:58:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:58:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:58:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:58:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2024-09-15T00:58:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:58:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:58:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:58:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:58:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:58:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-15T00:58:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_22.status 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_22.status 2024-09-15T00:58:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-15T00:58:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_22.status 2024-09-15T00:58:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.status 2024-09-15T00:58:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-15T00:58:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-15T00:58:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 128 2024-09-15T00:58:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_22.status 2024-09-15T00:58:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_22.status 2024-09-15T00:58:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-15T00:58:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:58:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=128 2024-09-15T00:58:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:58:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.status 2024-09-15T00:58:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.status 2024-09-15T00:58:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-15T00:58:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=56 2024-09-15T00:58:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 128 = 56 2024-09-15T00:58:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:58:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.status 2024-09-15T00:58:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.status 2024-09-15T00:58:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-15T00:58:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 128 = '' 2024-09-15T00:58:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:58:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.status 2024-09-15T00:58:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.status 2024-09-15T00:58:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-15T00:58:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 128 = '' 2024-09-15T00:58:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:58:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.status 2024-09-15T00:58:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.status 2024-09-15T00:58:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-15T00:58:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=128 2024-09-15T00:58:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 128 = 128 2024-09-15T00:58:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-15T00:58:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:58:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-15T00:58:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_22': 2024-09-15T00:58:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:58:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:58:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:58:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e2ac80e3d 2024-09-15T00:58:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e2ac80e3d 2024-09-15T00:58:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:26 2024 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:26 2024 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:26 2024 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 0559d020-e414-4263-981e-562438312202 2024-09-15T00:58:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:58:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-15T00:58:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T00:58:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:58:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_22.export 2024-09-15T00:58:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.export 2024-09-15T00:58:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_22.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.export 2024-09-15T00:58:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_22.export 2024-09-15T00:58:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.export 2024-09-15T00:58:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_22.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.export 2024-09-15T00:58:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_22.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_22.export 2024-09-15T00:58:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:58:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_23 2024-09-15T00:58:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-15T00:58:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:58:35.061 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8220814933118212) > weight (0.25) 2024-09-15T00:58:35.061 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-15T00:58:35.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-15T00:58:35.062 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-15T00:58:35.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T00:58:35.061+0000 7f84c5fcf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 42625) UID: 1000 2024-09-15T00:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-15T00:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-15T00:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:58:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-15T00:58:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-15T00:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:58:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:58:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:58:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:58:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:58:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:58:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:58:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:58:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-15T00:58:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-15T00:58:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:58:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:58:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:58:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-09-15T00:58:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-15T00:58:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:58:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:58:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:58:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-09-15T00:58:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:58:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:58:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:58:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:58:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:58:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:58:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:58:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-15T00:58:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:58:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:58:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-09-15T00:58:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-15T00:58:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_23.status 2024-09-15T00:58:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_23.status 2024-09-15T00:58:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-15T00:58:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_23.status 2024-09-15T00:58:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.status 2024-09-15T00:58:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-15T00:58:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-15T00:58:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 130 2024-09-15T00:58:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_23.status 2024-09-15T00:58:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_23.status 2024-09-15T00:58:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-15T00:58:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:58:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=130 2024-09-15T00:58:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:58:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.status 2024-09-15T00:58:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.status 2024-09-15T00:58:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-15T00:58:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=57 2024-09-15T00:58:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 130 = 57 2024-09-15T00:58:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:58:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.status 2024-09-15T00:58:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.status 2024-09-15T00:58:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-15T00:58:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 130 = '' 2024-09-15T00:58:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:58:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.status 2024-09-15T00:58:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.status 2024-09-15T00:58:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-15T00:58:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 130 = '' 2024-09-15T00:58:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:58:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.status 2024-09-15T00:58:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.status 2024-09-15T00:58:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-15T00:58:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=130 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 130 = 130 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:58:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-15T00:58:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_23': 2024-09-15T00:58:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:58:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:58:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:58:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126eae48d32f 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126eae48d32f 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:33 2024 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:33 2024 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:33 2024 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 80ac0c68-49b7-4ebc-8de0-5aae313e243b 2024-09-15T00:58:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:58:42.063 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-15T00:58:42.063 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:58:42.063 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-15T00:58:42.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-15T00:58:42.064 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-15T00:58:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-15T00:58:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T00:58:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:58:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_23.export 2024-09-15T00:58:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.export 2024-09-15T00:58:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_23.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.export 2024-09-15T00:58:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_23.export 2024-09-15T00:58:42.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-15T00:58:42.067 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-09-15T00:58:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.export 2024-09-15T00:58:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_23.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.export 2024-09-15T00:58:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_23.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_23.export 2024-09-15T00:58:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:58:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_24 2024-09-15T00:58:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-15T00:58:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-15T00:58:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:58:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:58:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:58:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:58:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-15T00:58:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:58:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:58:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:58:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:58:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:58:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:58:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:58:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-15T00:58:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-15T00:58:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-15T00:58:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:58:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:58:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:58:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-15T00:58:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-15T00:58:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2024-09-15T00:58:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:58:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:58:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:58:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:58:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2024-09-15T00:58:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:58:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:58:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:58:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:58:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:58:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:58:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:58:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-15T00:58:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:58:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-09-15T00:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-15T00:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_24.status 2024-09-15T00:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_24.status 2024-09-15T00:58:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-15T00:58:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_24.status 2024-09-15T00:58:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.status 2024-09-15T00:58:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-15T00:58:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-15T00:58:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 132 2024-09-15T00:58:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_24.status 2024-09-15T00:58:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_24.status 2024-09-15T00:58:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-15T00:58:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=132 2024-09-15T00:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:58:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.status 2024-09-15T00:58:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.status 2024-09-15T00:58:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-15T00:58:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=58 2024-09-15T00:58:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 132 = 58 2024-09-15T00:58:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:58:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.status 2024-09-15T00:58:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.status 2024-09-15T00:58:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-15T00:58:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 132 = '' 2024-09-15T00:58:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:58:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.status 2024-09-15T00:58:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.status 2024-09-15T00:58:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-15T00:58:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=132 2024-09-15T00:58:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 132 = 132 2024-09-15T00:58:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-15T00:58:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:58:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-15T00:58:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_24': 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e24f0aae 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e24f0aae 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:50:46 2024 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:50:46 2024 2024-09-15T00:58:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:50:46 2024 2024-09-15T00:58:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:58:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:58:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:58:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: a26c1dea-71ff-4c4c-ba15-84859056bc8f 2024-09-15T00:58:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:58:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-15T00:58:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T00:58:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:58:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_24.export 2024-09-15T00:58:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.export 2024-09-15T00:58:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_24.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.export 2024-09-15T00:58:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_24.export 2024-09-15T00:58:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.export 2024-09-15T00:58:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_24.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.export 2024-09-15T00:58:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_24.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_24.export 2024-09-15T00:58:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:58:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_25 2024-09-15T00:58:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-15T00:58:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:58:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-15T00:58:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-15T00:58:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-15T00:58:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:58:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:58:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:58:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-15T00:58:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:58:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:58:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:58:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:58:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:58:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:58:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:58:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:58:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-15T00:58:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:58:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:58:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:58:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:58:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:58:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:58:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:58:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-15T00:58:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-15T00:58:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-09-15T00:58:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:58:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:58:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:58:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:58:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:58:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:58:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:58:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:58:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:58:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:58:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:58:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:58:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-09-15T00:58:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-15T00:58:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:58:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:58:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-09-15T00:58:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:58:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:58:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:58:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:58:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-15T00:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-09-15T00:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-15T00:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_25.status 2024-09-15T00:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_25.status 2024-09-15T00:58:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-15T00:58:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_25.status 2024-09-15T00:58:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.status 2024-09-15T00:58:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-15T00:58:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-15T00:58:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 134 2024-09-15T00:58:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_25.status 2024-09-15T00:58:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_25.status 2024-09-15T00:58:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-15T00:58:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:58:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=134 2024-09-15T00:58:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:58:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.status 2024-09-15T00:58:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.status 2024-09-15T00:58:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-15T00:58:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=59 2024-09-15T00:58:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 134 = 59 2024-09-15T00:58:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:58:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.status 2024-09-15T00:58:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.status 2024-09-15T00:58:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-15T00:58:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:58:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:58:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:58:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:58:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 134 = '' 2024-09-15T00:58:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:58:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.status 2024-09-15T00:58:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.status 2024-09-15T00:58:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-15T00:58:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:58:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=134 2024-09-15T00:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 134 = 134 2024-09-15T00:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-15T00:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-15T00:58:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_25': 2024-09-15T00:58:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:58:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:58:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:58:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126ef601d665 2024-09-15T00:58:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126ef601d665 2024-09-15T00:58:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:00 2024 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:00 2024 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:00 2024 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 8c25e489-380c-471b-b67e-670250240dc3 2024-09-15T00:58:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:58:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-15T00:58:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T00:58:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:58:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_25.export 2024-09-15T00:58:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.export 2024-09-15T00:58:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_25.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.export 2024-09-15T00:58:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_25.export 2024-09-15T00:58:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.export 2024-09-15T00:58:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_25.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.export 2024-09-15T00:58:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_25.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_25.export 2024-09-15T00:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_26 2024-09-15T00:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-15T00:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:58:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-15T00:58:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:58:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-15T00:58:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:58:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:58:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:58:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:58:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-15T00:58:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:58:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:58:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:58:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:58:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:58:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:58:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-15T00:58:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:58:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:58:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:58:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:58:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:58:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:58:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:58:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:58:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:58:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-15T00:58:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-15T00:58:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-15T00:58:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:58:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:58:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:58:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:58:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-15T00:58:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-15T00:58:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:58:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:58:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:58:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:58:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:58:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:58:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:58:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:58:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:58:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:58:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-15T00:58:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:58:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:58:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:58:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:58:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:58:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:58:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-15T00:58:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_26.status 2024-09-15T00:58:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_26.status 2024-09-15T00:58:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-15T00:58:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_26.status 2024-09-15T00:58:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:58:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.status 2024-09-15T00:58:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-15T00:58:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:58:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-15T00:58:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 136 2024-09-15T00:58:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_26.status 2024-09-15T00:58:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:58:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:58:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:58:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_26.status 2024-09-15T00:58:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-15T00:58:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:58:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:58:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=136 2024-09-15T00:58:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:58:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:58:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:59:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.status 2024-09-15T00:59:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:59:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.status 2024-09-15T00:59:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-15T00:59:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=60 2024-09-15T00:59:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 136 = 60 2024-09-15T00:59:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:59:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.status 2024-09-15T00:59:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:59:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.status 2024-09-15T00:59:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-15T00:59:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 136 = '' 2024-09-15T00:59:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:59:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.status 2024-09-15T00:59:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:59:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.status 2024-09-15T00:59:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-15T00:59:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=136 2024-09-15T00:59:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 136 = 136 2024-09-15T00:59:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-15T00:59:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:59:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:59:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-15T00:59:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_26': 2024-09-15T00:59:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e8196fc3f 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e8196fc3f 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:13 2024 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:13 2024 2024-09-15T00:59:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:13 2024 2024-09-15T00:59:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:59:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:59:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:59:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 22c39bf0-0e09-4cf6-b50b-7c174804bc9e 2024-09-15T00:59:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:59:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-15T00:59:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T00:59:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:59:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_26.export 2024-09-15T00:59:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.export 2024-09-15T00:59:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_26.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.export 2024-09-15T00:59:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_26.export 2024-09-15T00:59:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.export 2024-09-15T00:59:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_26.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.export 2024-09-15T00:59:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_26.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_26.export 2024-09-15T00:59:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:59:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_27 2024-09-15T00:59:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-15T00:59:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:59:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-15T00:59:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-15T00:59:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:59:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-15T00:59:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:59:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:59:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:59:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-15T00:59:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:59:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:59:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:59:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:59:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:59:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:59:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:59:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:59:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:59:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:59:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:59:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:59:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:59:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:59:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:59:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-15T00:59:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-15T00:59:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-15T00:59:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:59:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:59:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:59:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-15T00:59:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-15T00:59:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-09-15T00:59:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:59:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:59:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:59:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:59:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:59:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:59:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:59:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:59:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-09-15T00:59:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:59:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:59:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:59:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:59:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:59:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:59:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-15T00:59:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_27.status 2024-09-15T00:59:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_27.status 2024-09-15T00:59:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-15T00:59:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_27.status 2024-09-15T00:59:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.status 2024-09-15T00:59:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-15T00:59:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-15T00:59:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 138 2024-09-15T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_27.status 2024-09-15T00:59:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_27.status 2024-09-15T00:59:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-15T00:59:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:59:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=138 2024-09-15T00:59:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:59:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.status 2024-09-15T00:59:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.status 2024-09-15T00:59:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-15T00:59:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=61 2024-09-15T00:59:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 138 = 61 2024-09-15T00:59:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.status 2024-09-15T00:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.status 2024-09-15T00:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-15T00:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 138 = '' 2024-09-15T00:59:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:59:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.status 2024-09-15T00:59:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.status 2024-09-15T00:59:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-15T00:59:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=138 2024-09-15T00:59:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 138 = 138 2024-09-15T00:59:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-15T00:59:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:59:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-15T00:59:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_27': 2024-09-15T00:59:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:59:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:59:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:59:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126eb5c3156d 2024-09-15T00:59:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126eb5c3156d 2024-09-15T00:59:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:19 2024 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:19 2024 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:19 2024 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 10250805-8ae3-4f07-92f0-130437cc82f8 2024-09-15T00:59:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-15T00:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T00:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_27.export 2024-09-15T00:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.export 2024-09-15T00:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_27.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.export 2024-09-15T00:59:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_27.export 2024-09-15T00:59:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.export 2024-09-15T00:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_27.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.export 2024-09-15T00:59:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_27.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_27.export 2024-09-15T00:59:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:59:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_28 2024-09-15T00:59:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-15T00:59:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:59:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-15T00:59:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-15T00:59:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:59:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:59:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:59:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:59:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:59:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-15T00:59:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-15T00:59:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:59:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:59:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:59:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-15T00:59:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-15T00:59:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2024-09-15T00:59:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:59:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:59:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:59:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:59:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2024-09-15T00:59:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:59:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:59:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:59:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:59:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:59:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-15T00:59:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_28.status 2024-09-15T00:59:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_28.status 2024-09-15T00:59:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-15T00:59:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_28.status 2024-09-15T00:59:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.status 2024-09-15T00:59:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-15T00:59:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-15T00:59:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 140 2024-09-15T00:59:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_28.status 2024-09-15T00:59:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_28.status 2024-09-15T00:59:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-15T00:59:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:59:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=140 2024-09-15T00:59:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:59:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.status 2024-09-15T00:59:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.status 2024-09-15T00:59:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-15T00:59:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=62 2024-09-15T00:59:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 140 = 62 2024-09-15T00:59:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:59:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.status 2024-09-15T00:59:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.status 2024-09-15T00:59:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-15T00:59:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 140 = '' 2024-09-15T00:59:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:59:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.status 2024-09-15T00:59:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.status 2024-09-15T00:59:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-15T00:59:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 140 = '' 2024-09-15T00:59:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:59:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.status 2024-09-15T00:59:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.status 2024-09-15T00:59:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-15T00:59:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=140 2024-09-15T00:59:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 140 = 140 2024-09-15T00:59:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-15T00:59:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:59:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_28': 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e3add4b45 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e3add4b45 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:31 2024 2024-09-15T00:59:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:31 2024 2024-09-15T00:59:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:31 2024 2024-09-15T00:59:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:59:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:59:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:59:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 5806c03a-ef8d-41cb-80aa-38c353b9ac79 2024-09-15T00:59:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:59:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-15T00:59:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T00:59:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:59:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_28.export 2024-09-15T00:59:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.export 2024-09-15T00:59:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_28.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.export 2024-09-15T00:59:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_28.export 2024-09-15T00:59:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.export 2024-09-15T00:59:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_28.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.export 2024-09-15T00:59:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_28.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_28.export 2024-09-15T00:59:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:59:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_29 2024-09-15T00:59:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-15T00:59:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:59:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-15T00:59:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-15T00:59:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:59:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-15T00:59:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:59:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:59:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:59:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-15T00:59:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:59:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:59:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:59:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:59:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:59:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:59:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:59:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:59:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:59:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-15T00:59:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-15T00:59:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-09-15T00:59:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:59:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:59:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:59:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:59:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-09-15T00:59:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-15T00:59:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:59:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:59:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-09-15T00:59:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:59:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:59:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:59:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:59:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:59:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:59:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:59:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-15T00:59:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:59:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:59:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-09-15T00:59:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-15T00:59:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_29.status 2024-09-15T00:59:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_29.status 2024-09-15T00:59:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-15T00:59:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_29.status 2024-09-15T00:59:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.status 2024-09-15T00:59:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-15T00:59:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-15T00:59:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 142 2024-09-15T00:59:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_29.status 2024-09-15T00:59:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_29.status 2024-09-15T00:59:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-15T00:59:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:59:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=142 2024-09-15T00:59:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:59:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.status 2024-09-15T00:59:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.status 2024-09-15T00:59:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-15T00:59:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=63 2024-09-15T00:59:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 142 = 63 2024-09-15T00:59:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:59:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.status 2024-09-15T00:59:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.status 2024-09-15T00:59:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-15T00:59:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 142 = '' 2024-09-15T00:59:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:59:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.status 2024-09-15T00:59:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.status 2024-09-15T00:59:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-15T00:59:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=142 2024-09-15T00:59:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 142 = 142 2024-09-15T00:59:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-15T00:59:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:59:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-15T00:59:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_29': 2024-09-15T00:59:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:59:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:59:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:59:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126eae3e0ea2 2024-09-15T00:59:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126eae3e0ea2 2024-09-15T00:59:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:59:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:59:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:59:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:59:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:37 2024 2024-09-15T00:59:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:37 2024 2024-09-15T00:59:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:37 2024 2024-09-15T00:59:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:59:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:59:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:59:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: b144ab42-9839-4e90-9820-f5ac9d0bf688 2024-09-15T00:59:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:59:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-15T00:59:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T00:59:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:59:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_29.export 2024-09-15T00:59:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.export 2024-09-15T00:59:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_29.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.export 2024-09-15T00:59:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_29.export 2024-09-15T00:59:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.export 2024-09-15T00:59:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_29.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.export 2024-09-15T00:59:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_29.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_29.export 2024-09-15T00:59:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:59:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_30 2024-09-15T00:59:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-15T00:59:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:59:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-15T00:59:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-15T00:59:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:59:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-15T00:59:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:59:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:59:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:59:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-15T00:59:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:59:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:59:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:59:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:59:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:59:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:59:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:59:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:59:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-15T00:59:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:59:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:59:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:59:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:59:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:59:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:59:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:59:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:59:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-15T00:59:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-15T00:59:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:59:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:59:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:59:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-09-15T00:59:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-15T00:59:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2024-09-15T00:59:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:59:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:59:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:59:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2024-09-15T00:59:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:59:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:59:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:59:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:59:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:59:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:59:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:59:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-15T00:59:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:59:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:59:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-09-15T00:59:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_30.status 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_30.status 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_30.status 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.status 2024-09-15T00:59:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-15T00:59:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-15T00:59:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 144 2024-09-15T00:59:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_30.status 2024-09-15T00:59:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_30.status 2024-09-15T00:59:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-15T00:59:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:59:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=144 2024-09-15T00:59:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:59:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.status 2024-09-15T00:59:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.status 2024-09-15T00:59:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-15T00:59:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=64 2024-09-15T00:59:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 144 = 64 2024-09-15T00:59:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:59:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.status 2024-09-15T00:59:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.status 2024-09-15T00:59:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-15T00:59:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 144 = '' 2024-09-15T00:59:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:59:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.status 2024-09-15T00:59:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.status 2024-09-15T00:59:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-15T00:59:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=144 2024-09-15T00:59:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 144 = 144 2024-09-15T00:59:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-15T00:59:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:59:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-15T00:59:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_30': 2024-09-15T00:59:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:59:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:59:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:59:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126ec47150cf 2024-09-15T00:59:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126ec47150cf 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:44 2024 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:44 2024 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:44 2024 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 5efc243e-6384-4dc5-97d4-ee66a1a5f7ec 2024-09-15T00:59:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:59:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-15T00:59:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T00:59:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:59:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_30.export 2024-09-15T00:59:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.export 2024-09-15T00:59:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_30.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.export 2024-09-15T00:59:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_30.export 2024-09-15T00:59:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.export 2024-09-15T00:59:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_30.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.export 2024-09-15T00:59:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_30.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_30.export 2024-09-15T00:59:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:59:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_31 2024-09-15T00:59:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-15T00:59:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:59:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-15T00:59:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-15T00:59:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:59:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-15T00:59:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:59:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:59:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:59:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-15T00:59:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:59:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:59:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:59:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:59:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:59:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:59:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:59:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:59:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:59:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:59:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:59:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-15T00:59:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-15T00:59:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:59:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:59:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:59:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:59:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-09-15T00:59:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-15T00:59:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2024-09-15T00:59:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:59:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:59:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:59:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:59:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2024-09-15T00:59:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:59:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:59:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:59:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:59:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:59:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-15T00:59:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_31.status 2024-09-15T00:59:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_31.status 2024-09-15T00:59:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-15T00:59:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_31.status 2024-09-15T00:59:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.status 2024-09-15T00:59:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-15T00:59:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-15T00:59:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 146 2024-09-15T00:59:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_31.status 2024-09-15T00:59:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_31.status 2024-09-15T00:59:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-15T00:59:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:59:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=146 2024-09-15T00:59:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:59:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.status 2024-09-15T00:59:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.status 2024-09-15T00:59:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-15T00:59:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=65 2024-09-15T00:59:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 146 = 65 2024-09-15T00:59:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:59:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.status 2024-09-15T00:59:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.status 2024-09-15T00:59:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-15T00:59:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 146 = '' 2024-09-15T00:59:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:59:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.status 2024-09-15T00:59:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.status 2024-09-15T00:59:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-15T00:59:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 146 = '' 2024-09-15T00:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:59:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.status 2024-09-15T00:59:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.status 2024-09-15T00:59:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-15T00:59:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=146 2024-09-15T00:59:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 146 = 146 2024-09-15T00:59:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-15T00:59:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:59:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-15T00:59:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_31': 2024-09-15T00:59:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:59:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:59:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:59:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e901edccc 2024-09-15T00:59:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e901edccc 2024-09-15T00:59:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:59:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:59:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:59:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:59:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:51:51 2024 2024-09-15T00:59:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:51:51 2024 2024-09-15T00:59:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:51:51 2024 2024-09-15T00:59:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:59:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:59:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:59:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 817ee7fa-f621-4e9d-a55c-56e846b6b7f5 2024-09-15T00:59:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:59:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-15T00:59:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T00:59:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:59:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_31.export 2024-09-15T00:59:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.export 2024-09-15T00:59:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_31.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.export 2024-09-15T00:59:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_31.export 2024-09-15T00:59:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.export 2024-09-15T00:59:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_31.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.export 2024-09-15T00:59:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_31.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_31.export 2024-09-15T00:59:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:59:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_32 2024-09-15T00:59:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-15T00:59:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:59:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-15T00:59:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-15T00:59:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:59:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-15T00:59:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:59:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:59:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:59:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-15T00:59:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:59:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:59:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:59:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:59:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:59:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:59:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:59:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:59:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:59:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:59:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:59:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-15T00:59:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-15T00:59:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-15T00:59:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:59:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:59:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:59:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:59:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:59:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:59:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:59:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-15T00:59:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-15T00:59:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:59:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:59:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:59:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2024-09-15T00:59:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:59:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:59:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:59:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:59:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:59:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:59:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-15T00:59:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_32.status 2024-09-15T00:59:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_32.status 2024-09-15T00:59:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-15T00:59:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_32.status 2024-09-15T00:59:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.status 2024-09-15T00:59:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-15T00:59:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-15T00:59:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 148 2024-09-15T00:59:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_32.status 2024-09-15T00:59:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_32.status 2024-09-15T00:59:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-15T00:59:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:59:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=148 2024-09-15T00:59:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:59:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.status 2024-09-15T00:59:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.status 2024-09-15T00:59:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-15T00:59:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=66 2024-09-15T00:59:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 148 = 66 2024-09-15T00:59:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:59:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.status 2024-09-15T00:59:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.status 2024-09-15T00:59:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-15T00:59:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 148 = '' 2024-09-15T00:59:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:59:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.status 2024-09-15T00:59:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.status 2024-09-15T00:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-15T00:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 148 = '' 2024-09-15T00:59:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:59:46.088 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.086+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbb65d790 2024-09-15T00:59:46.088 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.086+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbb8ea8f0 2024-09-15T00:59:46.088 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.087+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbb8ebe10 2024-09-15T00:59:46.090 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.087+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbbd3e5b0 2024-09-15T00:59:46.090 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.087+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde492b0 2024-09-15T00:59:46.090 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.088+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde49380 2024-09-15T00:59:46.090 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.088+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde495f0 2024-09-15T00:59:46.090 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.088+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde496c0 2024-09-15T00:59:46.090 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.088+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde49930 2024-09-15T00:59:46.090 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.088+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbf0c7380 2024-09-15T00:59:46.090 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.088+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbf3b0000 2024-09-15T00:59:46.090 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.088+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbfd59860 2024-09-15T00:59:46.090 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.088+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbff655f0 2024-09-15T00:59:46.091 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.089+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc00ed1e0 2024-09-15T00:59:46.091 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.089+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc0107930 2024-09-15T00:59:46.091 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.089+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc042a680 2024-09-15T00:59:46.091 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T00:59:46.089+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc04b7d40 2024-09-15T00:59:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.status 2024-09-15T00:59:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.status 2024-09-15T00:59:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-15T00:59:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=148 2024-09-15T00:59:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 148 = 148 2024-09-15T00:59:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-15T00:59:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:59:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-15T00:59:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_32': 2024-09-15T00:59:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:59:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:59:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e3da4ae76 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e3da4ae76 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:00 2024 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:00 2024 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:00 2024 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:59:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:59:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:59:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: cf355f92-98e8-41b9-8a17-84b24e81d482 2024-09-15T00:59:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:59:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-15T00:59:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T00:59:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:59:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_32.export 2024-09-15T00:59:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.export 2024-09-15T00:59:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_32.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.export 2024-09-15T00:59:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_32.export 2024-09-15T00:59:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.export 2024-09-15T00:59:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_32.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.export 2024-09-15T00:59:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_32.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_32.export 2024-09-15T00:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_33 2024-09-15T00:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-15T00:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:59:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-15T00:59:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-15T00:59:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:59:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-15T00:59:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:59:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:59:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:59:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-15T00:59:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:59:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:59:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:59:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:59:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:59:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:59:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:59:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:59:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-15T00:59:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:59:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:59:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:59:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:59:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:59:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:59:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:59:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:59:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-15T00:59:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-15T00:59:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:59:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-15T00:59:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-15T00:59:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-15T00:59:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:59:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:59:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:59:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:59:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:59:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:59:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-15T00:59:50.859 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-15T00:59:50.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-15T00:59:50.860 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-15T00:59:50.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T00:59:50.859+0000 7fab04fd6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 49611) UID: 1000 2024-09-15T00:59:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:59:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:59:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:59:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:59:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:59:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:59:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:59:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-15T00:59:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:59:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:59:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-09-15T00:59:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-15T00:59:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_33.status 2024-09-15T00:59:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_33.status 2024-09-15T00:59:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-15T00:59:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_33.status 2024-09-15T00:59:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.status 2024-09-15T00:59:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-15T00:59:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-15T00:59:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 150 2024-09-15T00:59:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_33.status 2024-09-15T00:59:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_33.status 2024-09-15T00:59:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-15T00:59:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:59:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=150 2024-09-15T00:59:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:59:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.status 2024-09-15T00:59:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.status 2024-09-15T00:59:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-15T00:59:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=67 2024-09-15T00:59:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 150 = 67 2024-09-15T00:59:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T00:59:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.status 2024-09-15T00:59:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.status 2024-09-15T00:59:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-15T00:59:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 150 = '' 2024-09-15T00:59:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T00:59:52.860 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-15T00:59:52.860 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T00:59:52.861 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-15T00:59:52.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-15T00:59:52.861 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-15T00:59:52.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-15T00:59:52.864 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-09-15T00:59:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.status 2024-09-15T00:59:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.status 2024-09-15T00:59:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-15T00:59:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T00:59:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T00:59:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T00:59:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T00:59:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 150 = '' 2024-09-15T00:59:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T00:59:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.status 2024-09-15T00:59:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.status 2024-09-15T00:59:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-15T00:59:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=150 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 150 = 150 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T00:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_33': 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126ee811aad5 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126ee811aad5 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:13 2024 2024-09-15T00:59:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:13 2024 2024-09-15T00:59:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:13 2024 2024-09-15T00:59:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T00:59:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T00:59:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T00:59:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 81151a5e-5338-4e8e-8ee9-a1dc81c31cc6 2024-09-15T00:59:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T00:59:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-15T00:59:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T00:59:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T00:59:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_33.export 2024-09-15T00:59:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.export 2024-09-15T00:59:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_33.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.export 2024-09-15T00:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_33.export 2024-09-15T00:59:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.export 2024-09-15T00:59:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T00:59:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_33.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.export 2024-09-15T00:59:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_33.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_33.export 2024-09-15T00:59:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T00:59:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_34 2024-09-15T00:59:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-15T00:59:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:59:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T00:59:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T00:59:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T00:59:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-15T00:59:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:59:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T00:59:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T00:59:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T00:59:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-15T00:59:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T00:59:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T00:59:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T00:59:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T00:59:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T00:59:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T00:59:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T00:59:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T00:59:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T00:59:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-15T00:59:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T00:59:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T00:59:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T00:59:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T00:59:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T00:59:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T00:59:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T00:59:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T00:59:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T00:59:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T00:59:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-15T00:59:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-15T00:59:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-15T00:59:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T00:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T00:59:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T00:59:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T00:59:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-15T00:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-15T00:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T00:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T00:59:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T00:59:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2024-09-15T00:59:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T00:59:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T00:59:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T00:59:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T00:59:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T00:59:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T00:59:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T00:59:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T00:59:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2024-09-15T00:59:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T00:59:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T00:59:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T00:59:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T00:59:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T00:59:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-09-15T00:59:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T00:59:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:59:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-15T00:59:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_34.status 2024-09-15T00:59:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_34.status 2024-09-15T00:59:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-15T00:59:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_34.status 2024-09-15T00:59:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T00:59:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.status 2024-09-15T00:59:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-15T00:59:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:59:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-15T00:59:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 152 2024-09-15T00:59:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_34.status 2024-09-15T00:59:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T00:59:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:59:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_34.status 2024-09-15T00:59:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-15T00:59:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T00:59:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=152 2024-09-15T00:59:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T00:59:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T00:59:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.status 2024-09-15T00:59:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T00:59:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T00:59:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T00:59:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.status 2024-09-15T00:59:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-15T00:59:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T00:59:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T00:59:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=68 2024-09-15T00:59:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 152 = 68 2024-09-15T00:59:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T00:59:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:00:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.status 2024-09-15T01:00:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T01:00:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.status 2024-09-15T01:00:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-15T01:00:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 152 = '' 2024-09-15T01:00:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:00:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.status 2024-09-15T01:00:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T01:00:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.status 2024-09-15T01:00:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-15T01:00:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 152 = '' 2024-09-15T01:00:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T01:00:02.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.status 2024-09-15T01:00:02.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:02.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:02.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T01:00:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.status 2024-09-15T01:00:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-15T01:00:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=152 2024-09-15T01:00:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 152 = 152 2024-09-15T01:00:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-15T01:00:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T01:00:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:00:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-15T01:00:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_34': 2024-09-15T01:00:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:00:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e19cf625f 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e19cf625f 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:20 2024 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:20 2024 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:20 2024 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:00:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:00:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: b1e366ef-a777-4914-8fb6-5426e0f998dd 2024-09-15T01:00:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:00:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-15T01:00:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T01:00:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:00:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_34.export 2024-09-15T01:00:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.export 2024-09-15T01:00:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_34.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.export 2024-09-15T01:00:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_34.export 2024-09-15T01:00:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.export 2024-09-15T01:00:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_34.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.export 2024-09-15T01:00:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_34.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_34.export 2024-09-15T01:00:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:00:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_35 2024-09-15T01:00:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-15T01:00:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:00:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-15T01:00:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-15T01:00:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:00:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-15T01:00:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:00:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:00:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:00:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-15T01:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:00:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-15T01:00:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:00:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:00:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:00:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:00:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:00:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:00:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:00:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:00:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-15T01:00:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-15T01:00:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2024-09-15T01:00:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:00:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:00:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:00:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:00:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:00:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:00:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:00:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:00:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:00:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:00:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:00:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:00:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2024-09-15T01:00:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-15T01:00:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2024-09-15T01:00:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:00:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:00:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:00:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:00:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2024-09-15T01:00:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:00:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:00:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:00:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:00:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:00:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:00:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:00:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-15T01:00:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_35.status 2024-09-15T01:00:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_35.status 2024-09-15T01:00:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-15T01:00:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_35.status 2024-09-15T01:00:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.status 2024-09-15T01:00:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-15T01:00:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-15T01:00:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 154 2024-09-15T01:00:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_35.status 2024-09-15T01:00:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_35.status 2024-09-15T01:00:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-15T01:00:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:00:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=154 2024-09-15T01:00:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:00:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.status 2024-09-15T01:00:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.status 2024-09-15T01:00:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-15T01:00:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=69 2024-09-15T01:00:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 154 = 69 2024-09-15T01:00:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:00:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.status 2024-09-15T01:00:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.status 2024-09-15T01:00:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-15T01:00:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 154 = '' 2024-09-15T01:00:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.status 2024-09-15T01:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.status 2024-09-15T01:00:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-15T01:00:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 154 = '' 2024-09-15T01:00:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T01:00:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.status 2024-09-15T01:00:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.status 2024-09-15T01:00:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-15T01:00:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=154 2024-09-15T01:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 154 = 154 2024-09-15T01:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-15T01:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:00:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-15T01:00:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_35': 2024-09-15T01:00:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:00:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:00:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e10469183 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e10469183 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:30 2024 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:30 2024 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:30 2024 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:00:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:00:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:00:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 264c4375-3155-4e86-9175-01a1a2ca5c03 2024-09-15T01:00:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:00:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-15T01:00:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:00:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:00:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_35.export 2024-09-15T01:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.export 2024-09-15T01:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_35.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.export 2024-09-15T01:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_35.export 2024-09-15T01:00:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.export 2024-09-15T01:00:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_35.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.export 2024-09-15T01:00:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_35.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_35.export 2024-09-15T01:00:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:00:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_36 2024-09-15T01:00:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-15T01:00:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:00:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-15T01:00:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-15T01:00:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:00:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-15T01:00:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:00:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:00:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:00:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-15T01:00:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:00:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:00:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:00:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:00:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:00:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:00:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:00:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:00:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:00:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:00:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:00:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:00:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:00:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:00:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-15T01:00:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-15T01:00:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:00:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:00:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:00:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-15T01:00:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-15T01:00:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-15T01:00:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:00:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:00:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:00:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:00:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:00:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:00:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:00:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:00:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:00:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:00:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:00:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:00:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-15T01:00:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:00:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:00:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:00:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:00:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:00:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:00:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:00:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-15T01:00:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_36.status 2024-09-15T01:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_36.status 2024-09-15T01:00:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-15T01:00:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_36.status 2024-09-15T01:00:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.status 2024-09-15T01:00:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-15T01:00:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-15T01:00:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 156 2024-09-15T01:00:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_36.status 2024-09-15T01:00:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_36.status 2024-09-15T01:00:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-15T01:00:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:00:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=156 2024-09-15T01:00:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:00:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.status 2024-09-15T01:00:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.status 2024-09-15T01:00:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-15T01:00:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=70 2024-09-15T01:00:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 156 = 70 2024-09-15T01:00:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:00:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.status 2024-09-15T01:00:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.status 2024-09-15T01:00:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-15T01:00:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 156 = '' 2024-09-15T01:00:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:00:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.status 2024-09-15T01:00:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.status 2024-09-15T01:00:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-15T01:00:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 156 = '' 2024-09-15T01:00:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T01:00:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.status 2024-09-15T01:00:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.status 2024-09-15T01:00:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-15T01:00:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=156 2024-09-15T01:00:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 156 = 156 2024-09-15T01:00:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-15T01:00:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:00:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:20.067 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9409244761025166) > weight (0.25) 2024-09-15T01:00:20.067 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5217167582901803) > weight (0.25) 2024-09-15T01:00:20.068 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.892340267649876) > weight (0.25) 2024-09-15T01:00:20.068 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.29267843898494705) > weight (0.25) 2024-09-15T01:00:20.068 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-15T01:00:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-15T01:00:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_36': 2024-09-15T01:00:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:00:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:00:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:00:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e24ef80bf 2024-09-15T01:00:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e24ef80bf 2024-09-15T01:00:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:43 2024 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:43 2024 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:43 2024 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: c97b3f87-8bd7-4173-9bd5-d9d0df48b80a 2024-09-15T01:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:00:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-15T01:00:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:00:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:00:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_36.export 2024-09-15T01:00:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.export 2024-09-15T01:00:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_36.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.export 2024-09-15T01:00:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_36.export 2024-09-15T01:00:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.export 2024-09-15T01:00:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_36.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.export 2024-09-15T01:00:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_36.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_36.export 2024-09-15T01:00:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:00:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_37 2024-09-15T01:00:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-15T01:00:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:00:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-15T01:00:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-15T01:00:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:00:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-15T01:00:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:00:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:00:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:00:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-15T01:00:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:00:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:00:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:00:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:00:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:00:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:00:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:00:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:00:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-15T01:00:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:00:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:00:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:00:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:00:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:00:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:00:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:00:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:00:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:00:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:00:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:00:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:00:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-15T01:00:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-15T01:00:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-09-15T01:00:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:00:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:00:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:00:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:00:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:00:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:00:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:00:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:00:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-09-15T01:00:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-15T01:00:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-09-15T01:00:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:00:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:00:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:00:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-09-15T01:00:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:00:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:00:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:00:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:00:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:00:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:00:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:00:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-15T01:00:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:00:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:00:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-09-15T01:00:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-15T01:00:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_37.status 2024-09-15T01:00:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_37.status 2024-09-15T01:00:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-15T01:00:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_37.status 2024-09-15T01:00:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.status 2024-09-15T01:00:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-15T01:00:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-15T01:00:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 158 2024-09-15T01:00:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_37.status 2024-09-15T01:00:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_37.status 2024-09-15T01:00:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-15T01:00:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:00:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=158 2024-09-15T01:00:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:00:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.status 2024-09-15T01:00:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.status 2024-09-15T01:00:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-15T01:00:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 158 = '' 2024-09-15T01:00:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:00:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.status 2024-09-15T01:00:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.status 2024-09-15T01:00:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-15T01:00:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 158 = '' 2024-09-15T01:00:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:00:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.status 2024-09-15T01:00:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.status 2024-09-15T01:00:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-15T01:00:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=158 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 158 = 158 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:00:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-15T01:00:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_37': 2024-09-15T01:00:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:00:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:00:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:00:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126eb1958d3c 2024-09-15T01:00:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126eb1958d3c 2024-09-15T01:00:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:00:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:00:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:00:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:00:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:52:58 2024 2024-09-15T01:00:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:52:58 2024 2024-09-15T01:00:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:52:58 2024 2024-09-15T01:00:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:00:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:00:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:00:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 45040e2c-2dd5-4e27-8546-d207a9f5ca1e 2024-09-15T01:00:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:00:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-15T01:00:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:00:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:00:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_37.export 2024-09-15T01:00:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.export 2024-09-15T01:00:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_37.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.export 2024-09-15T01:00:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_37.export 2024-09-15T01:00:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.export 2024-09-15T01:00:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_37.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.export 2024-09-15T01:00:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_37.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_37.export 2024-09-15T01:00:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:00:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_38 2024-09-15T01:00:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-15T01:00:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:00:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-15T01:00:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-15T01:00:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:00:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-15T01:00:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-15T01:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:00:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-15T01:00:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-15T01:00:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-15T01:00:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:00:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:00:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:00:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:00:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:00:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:00:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:00:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:00:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:00:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:00:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:00:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:00:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-15T01:00:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-15T01:00:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2024-09-15T01:00:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:00:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:00:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:00:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2024-09-15T01:00:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:00:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:00:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:00:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:00:31.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:00:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:00:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:00:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-15T01:00:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:00:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:00:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-09-15T01:00:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-15T01:00:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_38.status 2024-09-15T01:00:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_38.status 2024-09-15T01:00:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-15T01:00:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_38.status 2024-09-15T01:00:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.status 2024-09-15T01:00:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-15T01:00:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-15T01:00:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 160 2024-09-15T01:00:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_38.status 2024-09-15T01:00:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_38.status 2024-09-15T01:00:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-15T01:00:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:00:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=160 2024-09-15T01:00:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:00:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.status 2024-09-15T01:00:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.status 2024-09-15T01:00:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-15T01:00:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 160 = '' 2024-09-15T01:00:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:00:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.status 2024-09-15T01:00:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.status 2024-09-15T01:00:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-15T01:00:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 160 = '' 2024-09-15T01:00:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:00:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.status 2024-09-15T01:00:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.status 2024-09-15T01:00:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-15T01:00:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=160 2024-09-15T01:00:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 160 = 160 2024-09-15T01:00:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-15T01:00:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:00:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-15T01:00:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_38': 2024-09-15T01:00:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e7bcd449d 2024-09-15T01:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e7bcd449d 2024-09-15T01:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:53:13 2024 2024-09-15T01:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:53:13 2024 2024-09-15T01:00:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:53:13 2024 2024-09-15T01:00:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:00:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:00:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:00:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: d16734fd-d8ee-40b2-b689-bcc1dad3c46d 2024-09-15T01:00:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:00:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-15T01:00:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:00:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:00:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_38.export 2024-09-15T01:00:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.export 2024-09-15T01:00:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_38.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.export 2024-09-15T01:00:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_38.export 2024-09-15T01:00:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.export 2024-09-15T01:00:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_38.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.export 2024-09-15T01:00:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_38.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_38.export 2024-09-15T01:00:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:00:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_39 2024-09-15T01:00:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-15T01:00:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:00:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-15T01:00:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-15T01:00:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:00:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-15T01:00:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:00:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:00:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:00:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-15T01:00:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:00:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:00:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:00:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:00:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:00:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:00:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:00:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:00:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:00:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:00:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:00:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:00:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:00:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:00:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:00:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:00:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-15T01:00:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-15T01:00:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-15T01:00:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:00:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:00:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:00:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:00:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:00:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:00:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:00:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:00:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:00:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:00:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:00:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:00:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-15T01:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-15T01:00:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-09-15T01:00:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:00:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:00:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:00:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:00:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:00:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:00:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:00:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:00:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-09-15T01:00:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:00:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:00:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:00:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:00:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:00:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:00:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:00:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-15T01:00:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:00:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:00:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-09-15T01:00:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-15T01:00:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_39.status 2024-09-15T01:00:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_39.status 2024-09-15T01:00:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-15T01:00:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_39.status 2024-09-15T01:00:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.status 2024-09-15T01:00:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-15T01:00:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-15T01:00:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 162 2024-09-15T01:00:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_39.status 2024-09-15T01:00:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_39.status 2024-09-15T01:00:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-15T01:00:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:00:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=162 2024-09-15T01:00:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:00:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.status 2024-09-15T01:00:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.status 2024-09-15T01:00:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-15T01:00:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 162 = '' 2024-09-15T01:00:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:00:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.status 2024-09-15T01:00:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.status 2024-09-15T01:00:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-15T01:00:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 162 = '' 2024-09-15T01:00:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:00:40.175 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.171+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbbd3e5b0 2024-09-15T01:00:40.176 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.172+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde49380 2024-09-15T01:00:40.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde495f0 2024-09-15T01:00:40.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde496c0 2024-09-15T01:00:40.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbf0c7380 2024-09-15T01:00:40.180 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbf3b0000 2024-09-15T01:00:40.180 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbff655f0 2024-09-15T01:00:40.180 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc00ed1e0 2024-09-15T01:00:40.180 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc0107930 2024-09-15T01:00:40.180 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc042a680 2024-09-15T01:00:40.180 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:00:40.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc04b7d40 2024-09-15T01:00:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.status 2024-09-15T01:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.status 2024-09-15T01:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-15T01:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=162 2024-09-15T01:00:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 162 = 162 2024-09-15T01:00:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-15T01:00:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:00:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-15T01:00:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_39': 2024-09-15T01:00:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:00:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e4c170ab4 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e4c170ab4 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:53:25 2024 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:53:25 2024 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:53:25 2024 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:00:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:00:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 1dcd151a-a499-468b-b08a-ad4dd40aec05 2024-09-15T01:00:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:00:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-15T01:00:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:00:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:00:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_39.export 2024-09-15T01:00:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.export 2024-09-15T01:00:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_39.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.export 2024-09-15T01:00:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_39.export 2024-09-15T01:00:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.export 2024-09-15T01:00:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_39.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.export 2024-09-15T01:00:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_39.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_39.export 2024-09-15T01:00:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:00:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_40 2024-09-15T01:00:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-15T01:00:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:00:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-15T01:00:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-15T01:00:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:00:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-15T01:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:00:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-15T01:00:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:00:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:00:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:00:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:00:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:00:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:00:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:00:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:00:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-15T01:00:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:00:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:00:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:00:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:00:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:00:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:00:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:00:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:00:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-15T01:00:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-15T01:00:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:45.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-15T01:00:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:00:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:00:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:00:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:00:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:00:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:00:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:00:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:00:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-15T01:00:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-15T01:00:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2024-09-15T01:00:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:00:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:00:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:00:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:00:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:00:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:00:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:00:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:00:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:00:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:00:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:00:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:00:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2024-09-15T01:00:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:00:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:00:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:00:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:00:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-15T01:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:00:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:00:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-09-15T01:00:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-15T01:00:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_40.status 2024-09-15T01:00:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_40.status 2024-09-15T01:00:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-15T01:00:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_40.status 2024-09-15T01:00:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.status 2024-09-15T01:00:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-15T01:00:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-15T01:00:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 164 2024-09-15T01:00:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_40.status 2024-09-15T01:00:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_40.status 2024-09-15T01:00:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-15T01:00:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:00:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=164 2024-09-15T01:00:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:00:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.status 2024-09-15T01:00:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.status 2024-09-15T01:00:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-15T01:00:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 164 = '' 2024-09-15T01:00:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:00:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.status 2024-09-15T01:00:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.status 2024-09-15T01:00:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-15T01:00:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 164 = '' 2024-09-15T01:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:00:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.status 2024-09-15T01:00:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.status 2024-09-15T01:00:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-15T01:00:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=164 2024-09-15T01:00:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 164 = 164 2024-09-15T01:00:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-15T01:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-15T01:00:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_40': 2024-09-15T01:00:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:00:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:00:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:00:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e6452fca0 2024-09-15T01:00:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e6452fca0 2024-09-15T01:00:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:00:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:00:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:00:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:00:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:53:33 2024 2024-09-15T01:00:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:53:33 2024 2024-09-15T01:00:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:53:33 2024 2024-09-15T01:00:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:00:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:00:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:00:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: dae82d23-f158-4ac8-9d92-19f4212caa20 2024-09-15T01:00:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:00:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-15T01:00:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:00:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:00:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_40.export 2024-09-15T01:00:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.export 2024-09-15T01:00:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_40.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.export 2024-09-15T01:00:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_40.export 2024-09-15T01:00:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.export 2024-09-15T01:00:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_40.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.export 2024-09-15T01:00:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_40.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_40.export 2024-09-15T01:00:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:00:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_41 2024-09-15T01:00:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-15T01:00:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:00:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-15T01:00:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-15T01:00:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:00:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-15T01:00:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:00:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:00:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:00:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-15T01:00:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:00:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:00:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:00:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:00:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:00:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:00:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:00:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:00:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:00:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:00:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:00:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:00:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:00:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:00:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:00:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:00:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-15T01:00:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-15T01:00:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-15T01:00:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:00:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:00:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:00:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:00:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-15T01:00:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-15T01:00:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-15T01:00:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:00:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:00:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:00:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:00:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:00:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-15T01:00:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:00:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:00:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:00:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:00:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:00:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:00:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:00:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-15T01:00:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_41.status 2024-09-15T01:00:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_41.status 2024-09-15T01:00:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-15T01:00:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_41.status 2024-09-15T01:00:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.status 2024-09-15T01:00:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-15T01:00:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-15T01:00:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 166 2024-09-15T01:00:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_41.status 2024-09-15T01:00:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_41.status 2024-09-15T01:00:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-15T01:00:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:00:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=166 2024-09-15T01:00:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:00:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.status 2024-09-15T01:00:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.status 2024-09-15T01:00:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-15T01:00:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 166 = '' 2024-09-15T01:00:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:00:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.status 2024-09-15T01:00:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.status 2024-09-15T01:00:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-15T01:00:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:00:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:00:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:00:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:00:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 166 = '' 2024-09-15T01:00:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:00:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.status 2024-09-15T01:00:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.status 2024-09-15T01:00:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-15T01:00:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:00:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=166 2024-09-15T01:00:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 166 = 166 2024-09-15T01:00:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-15T01:00:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:00:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-15T01:00:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_41': 2024-09-15T01:00:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:00:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:00:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:00:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126ecf9aa634 2024-09-15T01:00:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126ecf9aa634 2024-09-15T01:00:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:00:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:00:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:00:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:00:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:53:41 2024 2024-09-15T01:00:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:53:41 2024 2024-09-15T01:00:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:53:41 2024 2024-09-15T01:00:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:00:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:00:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:00:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: d8226719-7493-4a0d-a661-b9bbd9dfb0fa 2024-09-15T01:00:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:00:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-15T01:00:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:00:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:00:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_41.export 2024-09-15T01:00:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.export 2024-09-15T01:00:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_41.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.export 2024-09-15T01:00:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_41.export 2024-09-15T01:00:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.export 2024-09-15T01:00:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:00:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_41.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.export 2024-09-15T01:00:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_41.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_41.export 2024-09-15T01:00:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:00:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_42 2024-09-15T01:00:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-15T01:00:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:00:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:00:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:00:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-15T01:00:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:00:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-15T01:00:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:00:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:00:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:00:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:00:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-15T01:00:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:00:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:00:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:00:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:00:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:00:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-15T01:00:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:00:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:00:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:00:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:00:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:00:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:00:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:00:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:00:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:00:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-15T01:00:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:00:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:00:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:00:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:00:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:00:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:00:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:00:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:00:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:00:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:00:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:00:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-15T01:00:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-15T01:00:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-15T01:00:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:00:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:00:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:00:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:00:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:00:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:00:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:00:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:00:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:00:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-15T01:00:59.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-15T01:00:59.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:00:59.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2024-09-15T01:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:00:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:00:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:00:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:00:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:00:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:00:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:00:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:00:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:00:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:00:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2024-09-15T01:00:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:00:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:00:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:00:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:00:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:00:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:00:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:00:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:00:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-15T01:00:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:00:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:00:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:00:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-09-15T01:00:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:00:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:00:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-15T01:00:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_42.status 2024-09-15T01:00:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_42.status 2024-09-15T01:00:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-15T01:00:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_42.status 2024-09-15T01:00:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:00:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.status 2024-09-15T01:00:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-15T01:00:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:00:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-15T01:00:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 168 2024-09-15T01:00:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_42.status 2024-09-15T01:00:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:00:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:00:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:00:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_42.status 2024-09-15T01:00:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-15T01:00:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:00:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:00:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=168 2024-09-15T01:00:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:00:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:00:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.status 2024-09-15T01:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.status 2024-09-15T01:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-15T01:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 168 = '' 2024-09-15T01:01:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:01:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.status 2024-09-15T01:01:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:01:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.status 2024-09-15T01:01:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-15T01:01:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 168 = '' 2024-09-15T01:01:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:01:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.status 2024-09-15T01:01:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.status 2024-09-15T01:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-15T01:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=168 2024-09-15T01:01:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 168 = 168 2024-09-15T01:01:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-15T01:01:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:01:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:01:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-15T01:01:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_42': 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e8f7ed0bc 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e8f7ed0bc 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:53:51 2024 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:53:51 2024 2024-09-15T01:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:53:51 2024 2024-09-15T01:01:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:01:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:01:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:01:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: c0403aa6-7690-4fb7-abc5-df8d17c3bf1b 2024-09-15T01:01:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:01:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-15T01:01:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:01:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:01:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_42.export 2024-09-15T01:01:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.export 2024-09-15T01:01:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_42.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.export 2024-09-15T01:01:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_42.export 2024-09-15T01:01:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.export 2024-09-15T01:01:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_42.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.export 2024-09-15T01:01:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_42.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_42.export 2024-09-15T01:01:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:01:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_43 2024-09-15T01:01:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-15T01:01:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:01:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-15T01:01:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-15T01:01:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:01:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-15T01:01:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:01:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:01:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:01:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-15T01:01:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:01:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:01:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:01:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:01:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:01:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:01:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:01:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:01:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-15T01:01:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:01:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:01:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:01:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:01:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:01:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-15T01:01:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-15T01:01:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-09-15T01:01:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:01:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:01:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:01:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:01:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-09-15T01:01:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-15T01:01:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-09-15T01:01:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:01:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:01:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:01:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:01:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:01:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:01:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:01:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:01:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-09-15T01:01:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:01:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:01:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:01:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:01:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:01:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:01:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:01:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-15T01:01:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_43.status 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_43.status 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-15T01:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_43.status 2024-09-15T01:01:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.status 2024-09-15T01:01:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-15T01:01:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-15T01:01:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 170 2024-09-15T01:01:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_43.status 2024-09-15T01:01:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_43.status 2024-09-15T01:01:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-15T01:01:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:01:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=170 2024-09-15T01:01:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:01:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.status 2024-09-15T01:01:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.status 2024-09-15T01:01:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-15T01:01:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=77 2024-09-15T01:01:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 170 = 77 2024-09-15T01:01:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:01:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.status 2024-09-15T01:01:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.status 2024-09-15T01:01:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-15T01:01:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 170 = '' 2024-09-15T01:01:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:01:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.status 2024-09-15T01:01:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.status 2024-09-15T01:01:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-15T01:01:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 170 = '' 2024-09-15T01:01:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T01:01:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.status 2024-09-15T01:01:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.status 2024-09-15T01:01:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-15T01:01:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=170 2024-09-15T01:01:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 170 = 170 2024-09-15T01:01:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-15T01:01:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:01:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-15T01:01:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_43': 2024-09-15T01:01:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:01:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:01:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e11e9374e 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e11e9374e 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:05 2024 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:05 2024 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:05 2024 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:01:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:01:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: d65182ef-5444-4d87-9ce1-616af2d50d7b 2024-09-15T01:01:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:01:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-15T01:01:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:01:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:01:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_43.export 2024-09-15T01:01:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.export 2024-09-15T01:01:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_43.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.export 2024-09-15T01:01:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_43.export 2024-09-15T01:01:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.export 2024-09-15T01:01:11.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_43.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.export 2024-09-15T01:01:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_43.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_43.export 2024-09-15T01:01:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:01:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_44 2024-09-15T01:01:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-15T01:01:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:01:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-15T01:01:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-15T01:01:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:01:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-15T01:01:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:01:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:01:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:01:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-15T01:01:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:01:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:01:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:01:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:01:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:01:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:01:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:01:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:01:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-15T01:01:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:01:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:01:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:01:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:01:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:01:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:01:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:01:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:01:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:01:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:01:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:01:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:01:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-15T01:01:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-15T01:01:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:01:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-15T01:01:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-15T01:01:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2024-09-15T01:01:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:01:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:01:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:01:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2024-09-15T01:01:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:01:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:01:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:01:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:01:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:01:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_44.status 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_44.status 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_44.status 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.status 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-15T01:01:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 172 2024-09-15T01:01:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_44.status 2024-09-15T01:01:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_44.status 2024-09-15T01:01:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-15T01:01:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:01:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=172 2024-09-15T01:01:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:01:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.status 2024-09-15T01:01:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.status 2024-09-15T01:01:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-15T01:01:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=78 2024-09-15T01:01:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 172 = 78 2024-09-15T01:01:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:01:15.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.status 2024-09-15T01:01:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.status 2024-09-15T01:01:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-15T01:01:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 172 = '' 2024-09-15T01:01:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:01:16.139 INFO:tasks.ceph.cluster1.osd.1.smithi002.stderr:2024-09-15T01:01:16.137+0000 7fc5d4903640 -1 reset not still connected to 0x55df07fd2c30 2024-09-15T01:01:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.status 2024-09-15T01:01:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.status 2024-09-15T01:01:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-15T01:01:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 172 = '' 2024-09-15T01:01:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T01:01:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.status 2024-09-15T01:01:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.status 2024-09-15T01:01:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-15T01:01:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=172 2024-09-15T01:01:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 172 = 172 2024-09-15T01:01:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-15T01:01:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:01:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-15T01:01:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_44': 2024-09-15T01:01:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:01:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:01:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:01:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e9f4350de 2024-09-15T01:01:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e9f4350de 2024-09-15T01:01:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:01:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:01:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:01:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:01:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:15 2024 2024-09-15T01:01:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:15 2024 2024-09-15T01:01:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:15 2024 2024-09-15T01:01:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:01:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:01:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:01:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: ba608aff-65b4-4bab-bcb7-e9adf2c42922 2024-09-15T01:01:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:01:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-15T01:01:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:01:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:01:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_44.export 2024-09-15T01:01:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.export 2024-09-15T01:01:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_44.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.export 2024-09-15T01:01:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_44.export 2024-09-15T01:01:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.export 2024-09-15T01:01:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_44.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.export 2024-09-15T01:01:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_44.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_44.export 2024-09-15T01:01:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:01:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_45 2024-09-15T01:01:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-15T01:01:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:01:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-15T01:01:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-15T01:01:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:01:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-15T01:01:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:01:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:01:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:01:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-15T01:01:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:01:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:01:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:01:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:01:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:01:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:01:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:01:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:01:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-15T01:01:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:01:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:01:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:01:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:01:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:01:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:01:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:01:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:01:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:01:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:01:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:01:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:01:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-15T01:01:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-15T01:01:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:01:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:01:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:01:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:01:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-15T01:01:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-15T01:01:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-09-15T01:01:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:01:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:01:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:01:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:01:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:01:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:01:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:01:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:01:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:01:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:01:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:01:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:01:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-09-15T01:01:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:01:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:01:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:01:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:01:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:01:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:01:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:01:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-15T01:01:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_45.status 2024-09-15T01:01:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_45.status 2024-09-15T01:01:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-15T01:01:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_45.status 2024-09-15T01:01:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.status 2024-09-15T01:01:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-15T01:01:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-15T01:01:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 174 2024-09-15T01:01:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_45.status 2024-09-15T01:01:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_45.status 2024-09-15T01:01:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-15T01:01:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:01:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=174 2024-09-15T01:01:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:01:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.status 2024-09-15T01:01:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.status 2024-09-15T01:01:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-15T01:01:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=79 2024-09-15T01:01:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 174 = 79 2024-09-15T01:01:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:01:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.status 2024-09-15T01:01:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.status 2024-09-15T01:01:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-15T01:01:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 174 = '' 2024-09-15T01:01:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:01:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.status 2024-09-15T01:01:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.status 2024-09-15T01:01:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-15T01:01:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=174 2024-09-15T01:01:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 174 = 174 2024-09-15T01:01:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-15T01:01:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:01:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-15T01:01:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_45': 2024-09-15T01:01:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:01:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126eadd6db78 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126eadd6db78 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:21 2024 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:21 2024 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:21 2024 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:01:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:01:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 7c98f381-5ad1-4de6-abca-0422679de2a8 2024-09-15T01:01:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:01:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-15T01:01:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:01:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:01:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_45.export 2024-09-15T01:01:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.export 2024-09-15T01:01:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_45.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.export 2024-09-15T01:01:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_45.export 2024-09-15T01:01:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.export 2024-09-15T01:01:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_45.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.export 2024-09-15T01:01:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_45.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_45.export 2024-09-15T01:01:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:01:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_46 2024-09-15T01:01:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-15T01:01:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:01:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:28.069 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5834934550820595) > weight (0.25) 2024-09-15T01:01:28.069 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.37804485649809594) > weight (0.25) 2024-09-15T01:01:28.070 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.916523401341944) > weight (0.25) 2024-09-15T01:01:28.070 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-15T01:01:28.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-15T01:01:28.070 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-15T01:01:28.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:01:28.070+0000 7f0943c2b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 40904) UID: 1000 2024-09-15T01:01:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-15T01:01:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:01:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:01:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:01:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-15T01:01:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:01:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:01:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:01:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:01:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:01:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:01:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:01:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:01:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-15T01:01:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:01:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:01:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:01:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:01:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:01:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:01:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:01:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:01:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-15T01:01:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-15T01:01:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-15T01:01:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:01:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:01:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:01:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:01:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:01:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:01:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:01:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-15T01:01:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-15T01:01:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-09-15T01:01:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:01:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:01:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:01:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:01:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:01:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:01:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:01:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:01:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:01:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:01:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:01:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:01:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-09-15T01:01:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:01:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:01:29.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:01:29.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:01:29.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:01:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:01:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:01:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-15T01:01:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:01:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:01:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-09-15T01:01:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-15T01:01:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_46.status 2024-09-15T01:01:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:29.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_46.status 2024-09-15T01:01:29.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-15T01:01:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_46.status 2024-09-15T01:01:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.status 2024-09-15T01:01:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-15T01:01:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-15T01:01:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 176 2024-09-15T01:01:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_46.status 2024-09-15T01:01:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_46.status 2024-09-15T01:01:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-15T01:01:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:01:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=176 2024-09-15T01:01:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:01:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.status 2024-09-15T01:01:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.status 2024-09-15T01:01:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-15T01:01:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=80 2024-09-15T01:01:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 176 = 80 2024-09-15T01:01:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:01:30.071 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-15T01:01:30.072 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:01:30.072 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-15T01:01:30.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-15T01:01:30.072 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-15T01:01:30.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-15T01:01:30.075 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-15T01:01:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.status 2024-09-15T01:01:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.status 2024-09-15T01:01:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-15T01:01:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 176 = '' 2024-09-15T01:01:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:01:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.status 2024-09-15T01:01:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.status 2024-09-15T01:01:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-15T01:01:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 176 = '' 2024-09-15T01:01:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T01:01:32.203 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:01:32.202+0000 7fc49cacc640 -1 reset not still connected to 0x560c5a0312b0 2024-09-15T01:01:32.203 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:01:32.202+0000 7fc49cacc640 -1 reset not still connected to 0x560c72f85ee0 2024-09-15T01:01:32.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe70f900 [3/8d62f872-96bf-4d6b-b93e-efad6f76511e] start: already running 2024-09-15T01:01:32.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe70fb80 [3/9a3b2ff6-9972-4b36-a5a7-25c8321572a0] start: already running 2024-09-15T01:01:32.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe740000 [3/a1914ce5-652f-465e-9c3f-e18ac5ac2ca8] start: already running 2024-09-15T01:01:32.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe740500 [3/b144ab42-9839-4e90-9820-f5ac9d0bf688] start: already running 2024-09-15T01:01:32.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe741900 [3/ba608aff-65b4-4bab-bcb7-e9adf2c42922] start: already running 2024-09-15T01:01:32.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe741180 [3/a26c1dea-71ff-4c4c-ba15-84859056bc8f] start: already running 2024-09-15T01:01:32.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe741400 [3/b1a838da-8633-4b6c-a717-7c58db5d3618] start: already running 2024-09-15T01:01:32.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe740780 [3/c0403aa6-7690-4fb7-abc5-df8d17c3bf1b] start: already running 2024-09-15T01:01:32.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe740f00 [3/d8226719-7493-4a0d-a661-b9bbd9dfb0fa] start: already running 2024-09-15T01:01:32.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe741b80 [3/cf355f92-98e8-41b9-8a17-84b24e81d482] start: already running 2024-09-15T01:01:32.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe77a000 [3/d16734fd-d8ee-40b2-b689-bcc1dad3c46d] start: already running 2024-09-15T01:01:32.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe77b400 [3/dae82d23-f158-4ac8-9d92-19f4212caa20] start: already running 2024-09-15T01:01:32.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe740c80 [3/d65182ef-5444-4d87-9ce1-616af2d50d7b] start: already running 2024-09-15T01:01:32.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe741680 [3/b1e366ef-a777-4914-8fb6-5426e0f998dd] start: already running 2024-09-15T01:01:32.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe77a280 [3/f6dc2e78-22b9-4458-a5da-18ecc500097e] start: already running 2024-09-15T01:01:32.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.217+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe740a00 [3/c97b3f87-8bd7-4173-9bd5-d9d0df48b80a] start: already running 2024-09-15T01:01:32.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:01:32.251+0000 7efc0fc70640 -1 rbd::mirror::ImageReplayer: 0x564abe740280 [3/ab51885a-03a7-4691-9f70-c8ead112d259] start: already running 2024-09-15T01:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.status 2024-09-15T01:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.status 2024-09-15T01:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-15T01:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=176 2024-09-15T01:01:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 176 = 176 2024-09-15T01:01:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-15T01:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-15T01:01:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_46': 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e9dbf278b 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e9dbf278b 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:33 2024 2024-09-15T01:01:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:33 2024 2024-09-15T01:01:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:33 2024 2024-09-15T01:01:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:01:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:01:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:01:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 6a06876f-8afd-4b45-82aa-fe83ff9bc561 2024-09-15T01:01:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:01:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-15T01:01:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:01:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:01:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_46.export 2024-09-15T01:01:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.export 2024-09-15T01:01:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_46.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.export 2024-09-15T01:01:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_46.export 2024-09-15T01:01:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.export 2024-09-15T01:01:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_46.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.export 2024-09-15T01:01:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_46.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_46.export 2024-09-15T01:01:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:01:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_47 2024-09-15T01:01:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-15T01:01:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:01:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-15T01:01:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-15T01:01:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:01:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-15T01:01:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:01:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:01:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:01:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-15T01:01:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:01:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:01:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:01:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:01:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:01:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:01:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:01:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-15T01:01:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-15T01:01:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-15T01:01:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:01:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:01:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:01:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:01:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:01:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:01:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:01:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:01:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:01:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:01:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:01:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:01:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-15T01:01:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-15T01:01:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-15T01:01:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:01:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:01:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:01:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:01:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:01:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:01:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:01:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:01:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:01:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:01:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:01:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:01:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-15T01:01:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:01:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:01:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:01:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:01:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-09-15T01:01:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-15T01:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_47.status 2024-09-15T01:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_47.status 2024-09-15T01:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-15T01:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_47.status 2024-09-15T01:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.status 2024-09-15T01:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-15T01:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-15T01:01:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 178 2024-09-15T01:01:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_47.status 2024-09-15T01:01:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_47.status 2024-09-15T01:01:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-15T01:01:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:01:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=178 2024-09-15T01:01:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:01:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.status 2024-09-15T01:01:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.status 2024-09-15T01:01:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-15T01:01:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=81 2024-09-15T01:01:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 178 = 81 2024-09-15T01:01:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:01:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.status 2024-09-15T01:01:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.status 2024-09-15T01:01:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-15T01:01:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 178 = '' 2024-09-15T01:01:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:01:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.status 2024-09-15T01:01:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.status 2024-09-15T01:01:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-15T01:01:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 178 = '' 2024-09-15T01:01:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T01:01:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.status 2024-09-15T01:01:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.status 2024-09-15T01:01:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-15T01:01:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=178 2024-09-15T01:01:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 178 = 178 2024-09-15T01:01:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-15T01:01:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:01:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-15T01:01:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_47': 2024-09-15T01:01:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:01:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:01:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:01:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e37b514d2 2024-09-15T01:01:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e37b514d2 2024-09-15T01:01:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:01:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:01:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:01:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:01:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:45 2024 2024-09-15T01:01:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:45 2024 2024-09-15T01:01:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:45 2024 2024-09-15T01:01:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:01:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:01:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:01:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 2f43fe62-b4f5-43a2-8094-d1a91f06de6f 2024-09-15T01:01:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:01:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-15T01:01:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:01:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:01:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_47.export 2024-09-15T01:01:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.export 2024-09-15T01:01:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_47.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.export 2024-09-15T01:01:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_47.export 2024-09-15T01:01:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.export 2024-09-15T01:01:42.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_47.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.export 2024-09-15T01:01:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_47.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_47.export 2024-09-15T01:01:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:01:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_48 2024-09-15T01:01:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-15T01:01:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:01:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:43.864 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9714755059751274) > weight (0.25) 2024-09-15T01:01:43.865 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-15T01:01:43.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-15T01:01:43.865 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-15T01:01:43.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:01:43.865+0000 7f00c2684640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 52157) UID: 1000 2024-09-15T01:01:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-15T01:01:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:01:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:01:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:01:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-15T01:01:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:01:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:01:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:01:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:01:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:01:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:01:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:01:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:01:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-15T01:01:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:01:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:01:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:01:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:01:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:01:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-15T01:01:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-15T01:01:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-15T01:01:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:01:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:01:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:01:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:01:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:01:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:01:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:01:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-15T01:01:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-15T01:01:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:01:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:01:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:01:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:01:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:01:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2024-09-15T01:01:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:01:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:01:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:01:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:01:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:01:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:01:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:01:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-15T01:01:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_48.status 2024-09-15T01:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_48.status 2024-09-15T01:01:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-15T01:01:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_48.status 2024-09-15T01:01:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.status 2024-09-15T01:01:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-15T01:01:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-15T01:01:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 180 2024-09-15T01:01:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_48.status 2024-09-15T01:01:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_48.status 2024-09-15T01:01:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-15T01:01:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:01:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=180 2024-09-15T01:01:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:01:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.status 2024-09-15T01:01:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.status 2024-09-15T01:01:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-15T01:01:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=82 2024-09-15T01:01:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 180 = 82 2024-09-15T01:01:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:01:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.status 2024-09-15T01:01:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.status 2024-09-15T01:01:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-15T01:01:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 180 = '' 2024-09-15T01:01:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:01:46.866 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-15T01:01:46.866 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:01:46.866 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-15T01:01:46.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-15T01:01:46.867 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-15T01:01:46.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-15T01:01:46.870 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-09-15T01:01:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.status 2024-09-15T01:01:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.status 2024-09-15T01:01:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-15T01:01:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=180 2024-09-15T01:01:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 180 = 180 2024-09-15T01:01:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-15T01:01:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:01:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-15T01:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_48': 2024-09-15T01:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e48b0656a 2024-09-15T01:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e48b0656a 2024-09-15T01:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:51 2024 2024-09-15T01:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:51 2024 2024-09-15T01:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:51 2024 2024-09-15T01:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 1bec5446-ddd9-4c25-9934-0781fdf9ac61 2024-09-15T01:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:01:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-15T01:01:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:01:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:01:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_48.export 2024-09-15T01:01:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.export 2024-09-15T01:01:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_48.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.export 2024-09-15T01:01:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_48.export 2024-09-15T01:01:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.export 2024-09-15T01:01:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_48.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.export 2024-09-15T01:01:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_48.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_48.export 2024-09-15T01:01:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:01:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_49 2024-09-15T01:01:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-15T01:01:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:01:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-15T01:01:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-15T01:01:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:01:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-15T01:01:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:01:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:01:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:01:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-15T01:01:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:01:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:01:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:01:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:01:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:01:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:01:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:01:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:01:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-15T01:01:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:01:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:01:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:01:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:01:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:01:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:01:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:01:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-15T01:01:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-15T01:01:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:01:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:01:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:01:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-09-15T01:01:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-15T01:01:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2024-09-15T01:01:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:01:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:01:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:01:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:01:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:01:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:01:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:01:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:01:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:01:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2024-09-15T01:01:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:01:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:01:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:01:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:01:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:01:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:01:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:01:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-15T01:01:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_49.status 2024-09-15T01:01:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_49.status 2024-09-15T01:01:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-15T01:01:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_49.status 2024-09-15T01:01:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:01:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.status 2024-09-15T01:01:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-15T01:01:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-15T01:01:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 182 2024-09-15T01:01:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_49.status 2024-09-15T01:01:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_49.status 2024-09-15T01:01:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-15T01:01:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:01:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=182 2024-09-15T01:01:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:01:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.status 2024-09-15T01:01:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.status 2024-09-15T01:01:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-15T01:01:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=83 2024-09-15T01:01:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 182 = 83 2024-09-15T01:01:53.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:53.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:01:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.status 2024-09-15T01:01:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.status 2024-09-15T01:01:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-15T01:01:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 182 = '' 2024-09-15T01:01:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:01:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.status 2024-09-15T01:01:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.status 2024-09-15T01:01:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-15T01:01:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:01:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:01:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:01:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:01:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:01:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 182 = '' 2024-09-15T01:01:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:01:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T01:01:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.status 2024-09-15T01:01:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.status 2024-09-15T01:01:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-15T01:01:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:01:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:01:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=182 2024-09-15T01:01:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 182 = 182 2024-09-15T01:01:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-15T01:01:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:01:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-15T01:01:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_49': 2024-09-15T01:01:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:01:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:01:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:01:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e6d205035 2024-09-15T01:01:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e6d205035 2024-09-15T01:01:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:01:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:01:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:01:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:01:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:54:57 2024 2024-09-15T01:01:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:54:57 2024 2024-09-15T01:01:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:54:57 2024 2024-09-15T01:01:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:01:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:01:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:01:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 3ecb23d9-a459-4c26-b283-80a7a224cdfb 2024-09-15T01:01:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:01:57.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:57.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-15T01:01:57.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:57.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:01:57.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:57.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:01:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_49.export 2024-09-15T01:01:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.export 2024-09-15T01:01:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_49.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.export 2024-09-15T01:01:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_49.export 2024-09-15T01:01:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.export 2024-09-15T01:01:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:01:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_49.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.export 2024-09-15T01:01:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_49.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_49.export 2024-09-15T01:01:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:01:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:01:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_50 2024-09-15T01:01:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-15T01:01:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:01:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:01:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:01:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-15T01:01:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-15T01:01:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:01:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-15T01:01:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:01:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=started 2024-09-15T01:01:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:01:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:01:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-15T01:01:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:01:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:01:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:01:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=started 2024-09-15T01:01:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_result 2024-09-15T01:01:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=stopped 2024-09-15T01:01:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-15T01:01:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_instance=cluster1 2024-09-15T01:01:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:01:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -i state 2024-09-15T01:01:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance=cluster1 2024-09-15T01:01:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local loop_instance 2024-09-15T01:01:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-15T01:01:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sleep 0 2024-09-15T01:01:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-15T01:01:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ seq 0 3 2024-09-15T01:01:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-15T01:01:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:0 2024-09-15T01:01:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-15T01:01:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:0 2024-09-15T01:01:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:01:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=0 2024-09-15T01:01:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-15T01:01:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:0 2024-09-15T01:01:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-15T01:01:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:0 2024-09-15T01:01:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=0 2024-09-15T01:01:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-15T01:01:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=0 2024-09-15T01:01:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=0 2024-09-15T01:01:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-15T01:01:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-15T01:01:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-15T01:01:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-15T01:01:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-15T01:01:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:1 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:1 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=1 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=1 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:1 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:1 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=1 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=1 2024-09-15T01:01:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=1 2024-09-15T01:01:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-15T01:01:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-15T01:01:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-15T01:01:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-15T01:01:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd mirror status : get status for rbd mirror 2024-09-15T01:01:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:admin_socket: invalid command 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1:2 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ shift 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local val=cluster1:2 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster_var_name=cluster 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local instance_var_name=instance 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:01:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=2 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval cluster=cluster1 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ eval instance=2 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local local_cluster=cluster1:2 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster=cluster1 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-15T01:01:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local val=cluster1:2 2024-09-15T01:01:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local cluster_var_name=local_cluster 2024-09-15T01:01:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ local instance_var_name=instance 2024-09-15T01:01:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ cluster=cluster1 2024-09-15T01:01:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ instance=2 2024-09-15T01:01:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-15T01:01:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval local_cluster=cluster1 2024-09-15T01:01:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ local_cluster=cluster1 2024-09-15T01:01:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ eval instance=2 2024-09-15T01:01:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ instance=2 2024-09-15T01:01:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-15T01:02:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:02:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:02:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-15T01:02:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-09-15T01:02:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ return 0 2024-09-15T01:02:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ status_result=' "state": "Replaying"' 2024-09-15T01:02:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo ' "state": "Replaying"' 2024-09-15T01:02:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ grep -i Replaying 2024-09-15T01:02:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: "state": "Replaying" 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=started 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test started = started 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = journal ']' 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' snapshot = snapshot ']' 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-09-15T01:02:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_cluster=cluster1 2024-09-15T01:02:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:02:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-15T01:02:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster2-mirror-image_50.status 2024-09-15T01:02:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:02:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local status_log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_50.status 2024-09-15T01:02:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-15T01:02:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ echo cluster1-mirror-image_50.status 2024-09-15T01:02:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-15T01:02:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local local_status_log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.status 2024-09-15T01:02:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-15T01:02:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:02:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-15T01:02:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Snapshot ID: 184 2024-09-15T01:02:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.nS85ynkalX/cluster2-mirror-image_50.status 2024-09-15T01:02:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:02:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_50.status 2024-09-15T01:02:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-15T01:02:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:02:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-15T01:02:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snapshot_id=184 2024-09-15T01:02:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:02:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:02:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.2 2024-09-15T01:02:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.status 2024-09-15T01:02:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:02:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:02:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.status 2024-09-15T01:02:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-15T01:02:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:02:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:02:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=84 2024-09-15T01:02:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 184 = 84 2024-09-15T01:02:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:02:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.4 2024-09-15T01:02:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.status 2024-09-15T01:02:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:02:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:02:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.status 2024-09-15T01:02:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-15T01:02:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:02:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:02:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:02:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:02:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:02:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:02:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 184 = '' 2024-09-15T01:02:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:02:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.8 2024-09-15T01:02:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.status 2024-09-15T01:02:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:02:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:02:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.status 2024-09-15T01:02:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-15T01:02:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:02:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:02:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:02:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:-:1.1: Document is empty 2024-09-15T01:02:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: 2024-09-15T01:02:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:^ 2024-09-15T01:02:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id= 2024-09-15T01:02:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 184 = '' 2024-09-15T01:02:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-15T01:02:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1.6 2024-09-15T01:02:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.status 2024-09-15T01:02:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:02:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:02:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local log=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.status 2024-09-15T01:02:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-15T01:02:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-15T01:02:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local primary_snapshot_id=184 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 184 = 184 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-15T01:02:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:02:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-15T01:02:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_50': 2024-09-15T01:02:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 128 MiB in 32 objects 2024-09-15T01:02:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:02:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 2 2024-09-15T01:02:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 126e818a4997 2024-09-15T01:02:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.126e818a4997 2024-09-15T01:02:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:02:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-15T01:02:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:02:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:02:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 00:55:05 2024 2024-09-15T01:02:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 00:55:05 2024 2024-09-15T01:02:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 00:55:05 2024 2024-09-15T01:02:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: protected: False 2024-09-15T01:02:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:02:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:02:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 5970ee5d-ad39-4b4f-828b-7d4f1adceca5 2024-09-15T01:02:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: false 2024-09-15T01:02:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-15T01:02:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:02:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap_name=snap 2024-09-15T01:02:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local ret=0 2024-09-15T01:02:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local rmt_export=/tmp/tmp.nS85ynkalX/cluster2-mirror-image_50.export 2024-09-15T01:02:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local loc_export=/tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.export 2024-09-15T01:02:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_50.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.export 2024-09-15T01:02:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.nS85ynkalX/cluster2-mirror-image_50.export 2024-09-15T01:02:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:02:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.export 2024-09-15T01:02:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-15T01:02:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cmp /tmp/tmp.nS85ynkalX/cluster2-mirror-image_50.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.export 2024-09-15T01:02:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -f /tmp/tmp.nS85ynkalX/cluster2-mirror-image_50.export /tmp/tmp.nS85ynkalX/cluster1-mirror-image_50.export 2024-09-15T01:02:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ testlog 'TEST: delete many images' 2024-09-15T01:02:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ tee -a /tmp/tmp.nS85ynkalX/rbd-mirror.test.log 2024-09-15T01:02:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ date '+%F %T' 2024-09-15T01:02:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo 2024-09-15 01:02:05 TEST: delete many images 2024-09-15T01:02:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:2024-09-15 01:02:05 TEST: delete many images 2024-09-15T01:02:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 50 2024-09-15T01:02:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_1 2024-09-15T01:02:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-15T01:02:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:02:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-15T01:02:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-15T01:02:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:02:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_1 2024-09-15T01:02:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:02:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-15T01:02:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-15T01:02:05.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:05.948+0000 7efc15c7c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abe21d860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:05.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:05.948+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe77a280 [3/f6dc2e78-22b9-4458-a5da-18ecc500097e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_2 2024-09-15T01:02:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-15T01:02:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:02:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-15T01:02:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-15T01:02:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:02:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_2 2024-09-15T01:02:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:02:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-15T01:02:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-15T01:02:08.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:08.978+0000 7efc15c7c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abe21da00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:08.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:08.978+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe672a00 [3/52f7a6a8-1ded-4b8e-b848-e1f2de6a62b1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:10.209 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:10.206+0000 7fc49cacc640 -1 reset not still connected to 0x560c53e39040 2024-09-15T01:02:10.214 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:10.206+0000 7fc49cacc640 -1 reset not still connected to 0x560c58b9c1a0 2024-09-15T01:02:10.214 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:10.209+0000 7fc49cacc640 -1 reset not still connected to 0x560c5c381860 2024-09-15T01:02:10.215 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:10.212+0000 7fc49cacc640 -1 reset not still connected to 0x560c62eb11e0 2024-09-15T01:02:10.215 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:10.212+0000 7fc49cacc640 -1 reset not still connected to 0x560c6c7ced00 2024-09-15T01:02:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_3 2024-09-15T01:02:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-15T01:02:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:02:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-15T01:02:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-15T01:02:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:02:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_3 2024-09-15T01:02:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:02:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-15T01:02:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-15T01:02:11.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:11.987+0000 7efc15c7c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abdad7520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:11.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:11.987+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe672280 [3/3262a16a-009b-44a2-b91f-62ecc486851b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_4 2024-09-15T01:02:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-15T01:02:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:02:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-15T01:02:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-15T01:02:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:02:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_4 2024-09-15T01:02:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:02:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-15T01:02:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-15T01:02:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:15.011+0000 7efc1647d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abd9a2b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:15.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:15.011+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe672f00 [3/51352989-1223-414e-86a9-443c3c234183] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_5 2024-09-15T01:02:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-15T01:02:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:02:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-15T01:02:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-15T01:02:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:02:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_5 2024-09-15T01:02:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:02:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-15T01:02:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-15T01:02:18.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:18.015+0000 7efc1647d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abe9bc1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:18.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:18.015+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe70f900 [3/8d62f872-96bf-4d6b-b93e-efad6f76511e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_6 2024-09-15T01:02:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-15T01:02:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:02:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-15T01:02:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-15T01:02:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:02:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_6 2024-09-15T01:02:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:02:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-15T01:02:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-15T01:02:21.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:21.036+0000 7efc1647d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abe9bc340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:21.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:21.036+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe256500 [3/003fd3d0-584f-46ad-a08c-185239d8066b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:22.164 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:22.162+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbbd3e5b0 2024-09-15T01:02:22.165 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:22.163+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde492b0 2024-09-15T01:02:22.165 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:22.163+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde49380 2024-09-15T01:02:22.165 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:22.163+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde495f0 2024-09-15T01:02:22.165 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:22.163+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde49930 2024-09-15T01:02:22.165 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:22.163+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbf3b0000 2024-09-15T01:02:22.165 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:22.163+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc00ed1e0 2024-09-15T01:02:22.165 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:22.163+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc0107930 2024-09-15T01:02:22.165 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:22.163+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc042a680 2024-09-15T01:02:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_7 2024-09-15T01:02:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-15T01:02:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:02:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-15T01:02:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-15T01:02:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:02:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_7 2024-09-15T01:02:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:02:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-15T01:02:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-15T01:02:24.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:24.038+0000 7efc1647d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abdad6820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:24.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:24.038+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe257900 [3/35b7b6ec-da56-4f63-91b0-35afde53e670] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_8 2024-09-15T01:02:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-15T01:02:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:02:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-15T01:02:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-15T01:02:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:02:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_8 2024-09-15T01:02:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:02:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-15T01:02:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-15T01:02:27.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:27.048+0000 7efc15c7c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abe9bc4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:27.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:27.048+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe673180 [3/5728982c-d76d-410e-bc93-a9b0b0701e25] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:28.881 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:28.879+0000 7fc49cacc640 -1 reset not still connected to 0x560c5a030750 2024-09-15T01:02:28.882 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:28.881+0000 7fc49cacc640 -1 reset not still connected to 0x560c5a0312b0 2024-09-15T01:02:28.885 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:28.883+0000 7fc49cacc640 -1 reset not still connected to 0x560c71a3ca90 2024-09-15T01:02:28.885 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:28.883+0000 7fc49cacc640 -1 reset not still connected to 0x560c72f85ee0 2024-09-15T01:02:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_9 2024-09-15T01:02:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-15T01:02:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:02:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-15T01:02:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-15T01:02:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:02:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_9 2024-09-15T01:02:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:02:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-15T01:02:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-15T01:02:30.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:30.018+0000 7efc15c7c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abe9bc680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:30.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:30.018+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe257680 [3/344e6c94-9331-4900-a47f-b1036d651c77] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:30.185 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:02:30.184+0000 7fc075401640 -1 reset not still connected to 0x55a0b860da00 2024-09-15T01:02:30.186 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:02:30.185+0000 7fc075401640 -1 reset not still connected to 0x55a0b86b1380 2024-09-15T01:02:30.186 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:02:30.185+0000 7fc075401640 -1 reset not still connected to 0x55a0b8965790 2024-09-15T01:02:30.187 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:02:30.185+0000 7fc075401640 -1 reset not still connected to 0x55a0b94fdee0 2024-09-15T01:02:30.187 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:02:30.185+0000 7fc075401640 -1 reset not still connected to 0x55a0b969a410 2024-09-15T01:02:30.187 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:02:30.185+0000 7fc075401640 -1 reset not still connected to 0x55a0b9b9b110 2024-09-15T01:02:30.187 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:02:30.186+0000 7fc075401640 -1 reset not still connected to 0x55a0bb2e9ee0 2024-09-15T01:02:30.187 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:02:30.186+0000 7fc075401640 -1 reset not still connected to 0x55a0bb3eb860 2024-09-15T01:02:30.187 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:02:30.186+0000 7fc075401640 -1 reset not still connected to 0x55a0bcbdcea0 2024-09-15T01:02:30.188 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:02:30.186+0000 7fc075401640 -1 reset not still connected to 0x55a0bdba52b0 2024-09-15T01:02:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_10 2024-09-15T01:02:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-15T01:02:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:02:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-15T01:02:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-15T01:02:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:02:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_10 2024-09-15T01:02:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:02:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-15T01:02:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-15T01:02:33.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:33.086+0000 7efc15c7c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abe9bc820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:33.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:33.086+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe70ef00 [3/7902b9a5-d221-4297-9894-a93a67ef6ff7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_11 2024-09-15T01:02:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-15T01:02:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T01:02:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-15T01:02:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-15T01:02:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T01:02:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_11 2024-09-15T01:02:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T01:02:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-15T01:02:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-15T01:02:36.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:36.108+0000 7efc15c7c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abe9bc9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:36.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:36.108+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe672780 [3/4d7f7b50-55a3-40e9-8e7e-87fb1c8cf94e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_12 2024-09-15T01:02:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-15T01:02:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T01:02:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-15T01:02:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-15T01:02:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T01:02:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_12 2024-09-15T01:02:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T01:02:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-15T01:02:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-15T01:02:39.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:39.113+0000 7efc15c7c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abe9bcb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:39.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:39.113+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe740000 [3/a1914ce5-652f-465e-9c3f-e18ac5ac2ca8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_13 2024-09-15T01:02:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-15T01:02:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T01:02:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-15T01:02:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-15T01:02:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T01:02:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_13 2024-09-15T01:02:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T01:02:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-15T01:02:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-15T01:02:42.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:42.123+0000 7efc1647d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abdad76c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:42.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:42.123+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe70e000 [3/66207760-d8d4-44f5-80a1-fdab006bbc16] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_14 2024-09-15T01:02:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-15T01:02:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T01:02:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-15T01:02:44.120 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.118+0000 7fc49cacc640 -1 reset not still connected to 0x560c573aa410 2024-09-15T01:02:44.125 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.122+0000 7fc49cacc640 -1 reset not still connected to 0x560c58b9c1a0 2024-09-15T01:02:44.125 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.122+0000 7fc49cacc640 -1 reset not still connected to 0x560c5a88a410 2024-09-15T01:02:44.125 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.122+0000 7fc49cacc640 -1 reset not still connected to 0x560c5d7e7110 2024-09-15T01:02:44.125 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.122+0000 7fc49cacc640 -1 reset not still connected to 0x560c5f7da410 2024-09-15T01:02:44.126 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.122+0000 7fc49cacc640 -1 reset not still connected to 0x560c5f905790 2024-09-15T01:02:44.126 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.122+0000 7fc49cacc640 -1 reset not still connected to 0x560c5fea91e0 2024-09-15T01:02:44.126 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.122+0000 7fc49cacc640 -1 reset not still connected to 0x560c611c9a00 2024-09-15T01:02:44.128 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.126+0000 7fc49cacc640 -1 reset not still connected to 0x560c61548d00 2024-09-15T01:02:44.128 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.126+0000 7fc49cacc640 -1 reset not still connected to 0x560c63abbe10 2024-09-15T01:02:44.128 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.126+0000 7fc49cacc640 -1 reset not still connected to 0x560c64b185b0 2024-09-15T01:02:44.128 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.126+0000 7fc49cacc640 -1 reset not still connected to 0x560c66b06340 2024-09-15T01:02:44.128 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:02:44.126+0000 7fc49cacc640 -1 reset not still connected to 0x560c6bbd3380 2024-09-15T01:02:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-15T01:02:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T01:02:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_14 2024-09-15T01:02:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T01:02:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-15T01:02:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-15T01:02:45.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:02:45.199+0000 7f172bc5a640 -1 rbd::mirror::ImageReplayer: 0x5601fac33900 [3/5a261e22-728a-47f5-ad1c-cec823ae7b9a] start: already running 2024-09-15T01:02:45.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:45.234+0000 7efc1647d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abdad7860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:45.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:45.234+0000 7efc1cc8a640 -1 rbd::mirror::ImageReplayer: 0x564abe673400 [3/5a261e22-728a-47f5-ad1c-cec823ae7b9a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_15 2024-09-15T01:02:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-15T01:02:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T01:02:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-15T01:02:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-15T01:02:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T01:02:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_15 2024-09-15T01:02:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T01:02:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-15T01:02:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-15T01:02:48.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:48.137+0000 7efc1647d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abd9a2b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:48.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:48.137+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe256780 [3/0df0e3a0-f71b-4bbc-8631-00ed1ea29cb7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:48.967 INFO:tasks.ceph.cluster2.osd.0.smithi059.stderr:2024-09-15T01:02:48.965+0000 7f173c535640 -1 reset not still connected to 0x5647114a24e0 2024-09-15T01:02:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_16 2024-09-15T01:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-15T01:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T01:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-15T01:02:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-15T01:02:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T01:02:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_16 2024-09-15T01:02:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T01:02:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-15T01:02:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-15T01:02:51.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:51.167+0000 7efc1647d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564abd9a29c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:51.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:51.167+0000 7efc1c489640 -1 rbd::mirror::ImageReplayer: 0x564abe741400 [3/b1a838da-8633-4b6c-a717-7c58db5d3618] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_17 2024-09-15T01:02:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-15T01:02:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T01:02:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-15T01:02:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-15T01:02:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T01:02:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_17 2024-09-15T01:02:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T01:02:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-15T01:02:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-15T01:02:54.077 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7973641182567843) > weight (0.25) 2024-09-15T01:02:54.077 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7905933264066929) > weight (0.25) 2024-09-15T01:02:54.077 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-15T01:02:54.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-15T01:02:54.078 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-15T01:02:54.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:02:54.077+0000 7efc1d48b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 49948) UID: 1000 2024-09-15T01:02:54.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:02:54.182+0000 7fd6b6ca7640 -1 rbd::mirror::ImageMap: 0x55cf9cbe8360 handle_peer_ack_remove: failed to remove global_image_id=9a3b2ff6-9972-4b36-a5a7-25c8321572a0 2024-09-15T01:02:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_18 2024-09-15T01:02:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-15T01:02:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T01:02:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-15T01:02:56.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:56.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbb8ea8f0 2024-09-15T01:02:56.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:56.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbb8ebe10 2024-09-15T01:02:56.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:56.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbbd3e5b0 2024-09-15T01:02:56.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:56.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde492b0 2024-09-15T01:02:56.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:56.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbde49930 2024-09-15T01:02:56.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:56.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecbf3b0000 2024-09-15T01:02:56.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:56.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc0107930 2024-09-15T01:02:56.178 INFO:tasks.ceph.cluster1.osd.0.smithi002.stderr:2024-09-15T01:02:56.174+0000 7fbbc68d9640 -1 reset not still connected to 0x55ecc042a680 2024-09-15T01:02:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-15T01:02:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T01:02:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_18 2024-09-15T01:02:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T01:02:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-15T01:02:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-15T01:02:57.078 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-15T01:02:57.078 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:02:57.079 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-15T01:02:57.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-15T01:02:57.079 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-15T01:02:57.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-15T01:02:57.082 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-15T01:02:58.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:02:58.158+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9cf0e680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:02:58.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:02:58.158+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d1b80 [3/8c63bcd3-d5d8-46ca-9685-256e72243858] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:02:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:02:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:02:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:02:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_19 2024-09-15T01:02:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-15T01:02:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T01:02:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:02:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-15T01:02:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-15T01:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T01:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:02:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_19 2024-09-15T01:02:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:02:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:02:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T01:02:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-15T01:02:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-15T01:03:00.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:00.024+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a7860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:00.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:00.024+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4b180 [3/237a420c-4d63-4519-a90b-522ac9d2b086] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_20 2024-09-15T01:03:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-15T01:03:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T01:03:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-15T01:03:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-15T01:03:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T01:03:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_20 2024-09-15T01:03:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T01:03:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-15T01:03:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-15T01:03:03.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:03.049+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8e0d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:03.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:03.049+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3e2280 [3/ab51885a-03a7-4691-9f70-c8ead112d259] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_21 2024-09-15T01:03:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-15T01:03:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T01:03:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-15T01:03:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-15T01:03:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T01:03:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_21 2024-09-15T01:03:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T01:03:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-15T01:03:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-15T01:03:06.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:06.054+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a7380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:06.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:06.054+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d0500 [3/69c7326f-a13b-465e-a7c0-8dec344e635a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_22 2024-09-15T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-15T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T01:03:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-15T01:03:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-15T01:03:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T01:03:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_22 2024-09-15T01:03:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T01:03:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-15T01:03:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-15T01:03:09.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:09.076+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8de4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:09.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:09.076+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4a500 [3/0559d020-e414-4263-981e-562438312202] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_23 2024-09-15T01:03:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-15T01:03:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T01:03:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-15T01:03:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-15T01:03:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T01:03:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_23 2024-09-15T01:03:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T01:03:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-15T01:03:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-15T01:03:12.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:12.081+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c6c7860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:12.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:12.081+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d1180 [3/80ac0c68-49b7-4ebc-8de0-5aae313e243b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_24 2024-09-15T01:03:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-15T01:03:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T01:03:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-15T01:03:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-15T01:03:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T01:03:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_24 2024-09-15T01:03:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T01:03:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-15T01:03:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-15T01:03:15.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:15.028+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8de820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:15.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:15.028+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3e2000 [3/a26c1dea-71ff-4c4c-ba15-84859056bc8f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_25 2024-09-15T01:03:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-15T01:03:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T01:03:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-15T01:03:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-15T01:03:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T01:03:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_25 2024-09-15T01:03:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T01:03:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-15T01:03:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-15T01:03:18.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:18.108+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a31e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:18.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:18.108+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d1900 [3/8c25e489-380c-471b-b67e-670250240dc3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_26 2024-09-15T01:03:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-15T01:03:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T01:03:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-15T01:03:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-15T01:03:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T01:03:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_26 2024-09-15T01:03:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T01:03:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-15T01:03:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-15T01:03:21.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:21.128+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a6680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:21.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:21.128+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4af00 [3/22c39bf0-0e09-4cf6-b50b-7c174804bc9e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_27 2024-09-15T01:03:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-15T01:03:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T01:03:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-15T01:03:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-15T01:03:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T01:03:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_27 2024-09-15T01:03:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T01:03:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-15T01:03:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-15T01:03:24.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:24.128+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a7a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:24.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:24.128+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4a780 [3/10250805-8ae3-4f07-92f0-130437cc82f8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_28 2024-09-15T01:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-15T01:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T01:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-15T01:03:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-15T01:03:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T01:03:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_28 2024-09-15T01:03:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T01:03:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-15T01:03:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-15T01:03:27.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:27.138+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9cf0eb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:27.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:27.138+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d0000 [3/5806c03a-ef8d-41cb-80aa-38c353b9ac79] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_29 2024-09-15T01:03:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-15T01:03:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T01:03:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-15T01:03:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-15T01:03:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T01:03:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_29 2024-09-15T01:03:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T01:03:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-15T01:03:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-15T01:03:30.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:30.026+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c6c76c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:30.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:30.026+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3e2500 [3/b144ab42-9839-4e90-9820-f5ac9d0bf688] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_30 2024-09-15T01:03:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-15T01:03:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T01:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-15T01:03:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-15T01:03:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T01:03:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_30 2024-09-15T01:03:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T01:03:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-15T01:03:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-15T01:03:33.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:33.160+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9cf0ed00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:33.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:33.160+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d0280 [3/5efc243e-6384-4dc5-97d4-ee66a1a5f7ec] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_31 2024-09-15T01:03:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-15T01:03:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T01:03:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-15T01:03:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-15T01:03:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T01:03:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_31 2024-09-15T01:03:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T01:03:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-15T01:03:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-15T01:03:36.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:36.177+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c6c7520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:36.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:36.177+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d1680 [3/817ee7fa-f621-4e9d-a55c-56e846b6b7f5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_32 2024-09-15T01:03:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-15T01:03:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T01:03:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-15T01:03:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-15T01:03:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T01:03:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_32 2024-09-15T01:03:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T01:03:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-15T01:03:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-15T01:03:39.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:39.176+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9cbf41a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:39.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:39.176+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3e3180 [3/cf355f92-98e8-41b9-8a17-84b24e81d482] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_33 2024-09-15T01:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-15T01:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T01:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-15T01:03:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-15T01:03:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T01:03:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_33 2024-09-15T01:03:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T01:03:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-15T01:03:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-15T01:03:42.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:42.191+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c6c7380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:42.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:42.191+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d1400 [3/81151a5e-5338-4e8e-8ee9-a1dc81c31cc6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:42.871 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-15T01:03:42.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-15T01:03:42.872 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-15T01:03:42.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:03:42.872+0000 7f0c8a530640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 58034) UID: 1000 2024-09-15T01:03:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_34 2024-09-15T01:03:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-15T01:03:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T01:03:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-15T01:03:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-15T01:03:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T01:03:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_34 2024-09-15T01:03:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T01:03:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-15T01:03:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-15T01:03:45.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:45.026+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9cbf4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:45.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:45.026+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3e2780 [3/b1e366ef-a777-4914-8fb6-5426e0f998dd] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_35 2024-09-15T01:03:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-15T01:03:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:03:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-15T01:03:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-15T01:03:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:03:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_35 2024-09-15T01:03:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:03:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-15T01:03:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-15T01:03:48.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:48.209+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5aa340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:48.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:48.209+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4b400 [3/264c4375-3155-4e86-9175-01a1a2ca5c03] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:49.873 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-15T01:03:49.874 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:03:49.874 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-15T01:03:49.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-15T01:03:49.874 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-15T01:03:49.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-15T01:03:49.878 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-15T01:03:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_36 2024-09-15T01:03:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-15T01:03:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:03:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-15T01:03:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-15T01:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_36 2024-09-15T01:03:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:03:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-15T01:03:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-15T01:03:51.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:51.225+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8e0d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:51.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:51.225+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3e2f00 [3/c97b3f87-8bd7-4173-9bd5-d9d0df48b80a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_37 2024-09-15T01:03:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-15T01:03:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:03:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-15T01:03:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-15T01:03:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:03:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_37 2024-09-15T01:03:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:03:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-15T01:03:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-15T01:03:54.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:54.231+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a3040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:54.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:54.231+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4bb80 [3/45040e2c-2dd5-4e27-8546-d207a9f5ca1e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_38 2024-09-15T01:03:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-15T01:03:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:03:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-15T01:03:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-15T01:03:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:03:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_38 2024-09-15T01:03:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:03:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-15T01:03:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-15T01:03:57.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:57.038+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a2d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:03:57.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:03:57.038+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3e3400 [3/d16734fd-d8ee-40b2-b689-bcc1dad3c46d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:03:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:03:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:03:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:03:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_39 2024-09-15T01:03:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-15T01:03:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:03:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:03:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-15T01:03:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:03:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-15T01:03:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:03:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:03:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_39 2024-09-15T01:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:03:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-15T01:03:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-15T01:04:00.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:00.037+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a76c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:00.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:00.037+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4ac80 [3/1dcd151a-a499-468b-b08a-ad4dd40aec05] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_40 2024-09-15T01:04:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-15T01:04:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:04:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-15T01:04:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-15T01:04:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:04:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_40 2024-09-15T01:04:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:04:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-15T01:04:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-15T01:04:03.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:03.065+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a31e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:03.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:03.065+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d462000 [3/dae82d23-f158-4ac8-9d92-19f4212caa20] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_41 2024-09-15T01:04:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-15T01:04:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:04:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-15T01:04:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-15T01:04:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:04:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_41 2024-09-15T01:04:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:04:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-15T01:04:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-15T01:04:06.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:06.076+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a2ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:06.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:06.076+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3e3b80 [3/d8226719-7493-4a0d-a661-b9bbd9dfb0fa] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_42 2024-09-15T01:04:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-15T01:04:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:04:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-15T01:04:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-15T01:04:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:04:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_42 2024-09-15T01:04:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:04:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-15T01:04:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-15T01:04:09.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:09.086+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8de820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:09.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:09.086+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3e2c80 [3/c0403aa6-7690-4fb7-abc5-df8d17c3bf1b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_43 2024-09-15T01:04:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-15T01:04:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:04:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-15T01:04:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-15T01:04:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:04:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_43 2024-09-15T01:04:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:04:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-15T01:04:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-15T01:04:12.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:12.103+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8de4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:12.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:12.103+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3e2a00 [3/d65182ef-5444-4d87-9ce1-616af2d50d7b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_44 2024-09-15T01:04:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-15T01:04:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:04:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-15T01:04:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-15T01:04:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:04:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:14.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_44 2024-09-15T01:04:14.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:14.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:04:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-15T01:04:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-15T01:04:15.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:15.039+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8e0d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:15.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:15.039+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d0a00 [3/ba608aff-65b4-4bab-bcb7-e9adf2c42922] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_45 2024-09-15T01:04:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-15T01:04:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:04:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-15T01:04:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-15T01:04:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:04:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_45 2024-09-15T01:04:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:04:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-15T01:04:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-15T01:04:18.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:18.126+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a3a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:18.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:18.126+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d0f00 [3/7c98f381-5ad1-4de6-abca-0422679de2a8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_46 2024-09-15T01:04:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-15T01:04:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:04:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-15T01:04:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-15T01:04:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:04:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_46 2024-09-15T01:04:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:04:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-15T01:04:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-15T01:04:21.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:21.144+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8e0ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:21.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:21.144+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d0780 [3/6a06876f-8afd-4b45-82aa-fe83ff9bc561] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_47 2024-09-15T01:04:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-15T01:04:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:04:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-15T01:04:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-15T01:04:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:04:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_47 2024-09-15T01:04:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:04:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-15T01:04:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-15T01:04:24.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:24.133+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8e1040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:24.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:24.133+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4b680 [3/2f43fe62-b4f5-43a2-8094-d1a91f06de6f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_48 2024-09-15T01:04:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-15T01:04:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:04:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-15T01:04:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-15T01:04:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:04:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_48 2024-09-15T01:04:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:04:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-15T01:04:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-15T01:04:27.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:27.153+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8e11e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:27.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:27.153+0000 7fd6b74a8640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4aa00 [3/1bec5446-ddd9-4c25-9934-0781fdf9ac61] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_49 2024-09-15T01:04:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-15T01:04:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:04:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-15T01:04:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-15T01:04:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:04:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_49 2024-09-15T01:04:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:04:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-15T01:04:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-15T01:04:30.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:30.050+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a2d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:30.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:30.050+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4b900 [3/3ecb23d9-a459-4c26-b283-80a7a224cdfb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_50 2024-09-15T01:04:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-15T01:04:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:04:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local snap=snap 2024-09-15T01:04:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-15T01:04:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing snap: 100% complete...done. 2024-09-15T01:04:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-15T01:04:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:04:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:04:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:04:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_50 2024-09-15T01:04:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:04:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-15T01:04:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-15T01:04:33.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:33.184+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c5a31e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:33.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:33.184+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d0c80 [3/5970ee5d-ad39-4b4f-828b-7d4f1adceca5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:04:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-15T01:04:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-15T01:04:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ tee -a /tmp/tmp.nS85ynkalX/rbd-mirror.test.log 2024-09-15T01:04:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ date '+%F %T' 2024-09-15T01:04:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo 2024-09-15 01:04:34 TEST: image deletions should propagate 2024-09-15T01:04:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:2024-09-15 01:04:34 TEST: image deletions should propagate 2024-09-15T01:04:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-15T01:04:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_count=0 2024-09-15T01:04:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local count 2024-09-15T01:04:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local last_count=0 2024-09-15T01:04:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ true 2024-09-15T01:04:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 40 2024-09-15T01:04:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in `seq 1 40` 2024-09-15T01:04:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 1 -ne 1 2024-09-15T01:04:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-15T01:04:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep 'images: ' 2024-09-15T01:04:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cut '-d ' -f 2 2024-09-15T01:04:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: failed to query services: (13) Permission denied 2024-09-15T01:04:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ count=0 2024-09-15T01:04:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 = 0 2024-09-15T01:04:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-15T01:04:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state 2024-09-15T01:04:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 40 2024-09-15T01:04:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in `seq 1 40` 2024-09-15T01:04:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 1 -ne 1 2024-09-15T01:04:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-15T01:04:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep 'image health:' 2024-09-15T01:04:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cut '-d ' -f 3 2024-09-15T01:04:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: failed to query services: (13) Permission denied 2024-09-15T01:04:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ state=OK 2024-09-15T01:04:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test OK = ERROR 2024-09-15T01:04:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test OK = OK 2024-09-15T01:04:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 50 2024-09-15T01:04:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_1 2024-09-15T01:04:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-15T01:04:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_1 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_1 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-15T01:04:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-15T01:04:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-15T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_1 2024-09-15T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_1 2024-09-15T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-15T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-15T01:04:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_2 2024-09-15T01:04:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-15T01:04:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:04:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_2 2024-09-15T01:04:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_2 2024-09-15T01:04:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-15T01:04:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-15T01:04:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-15T01:04:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_2 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_2 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-15T01:04:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-15T01:04:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_3 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_3 2024-09-15T01:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_3 2024-09-15T01:04:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-15T01:04:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-15T01:04:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-15T01:04:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:04:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_3 2024-09-15T01:04:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_3 2024-09-15T01:04:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-15T01:04:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-15T01:04:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_4 2024-09-15T01:04:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-15T01:04:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:04:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_4 2024-09-15T01:04:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_4 2024-09-15T01:04:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-15T01:04:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-15T01:04:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-15T01:04:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:04:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_4 2024-09-15T01:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_4 2024-09-15T01:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-15T01:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-15T01:04:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_5 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_5 2024-09-15T01:04:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_5 2024-09-15T01:04:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-15T01:04:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-15T01:04:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-15T01:04:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:04:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_5 2024-09-15T01:04:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_5 2024-09-15T01:04:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-15T01:04:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-15T01:04:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_6 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_6 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_6 2024-09-15T01:04:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-15T01:04:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-15T01:04:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-15T01:04:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:04:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_6 2024-09-15T01:04:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_6 2024-09-15T01:04:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-15T01:04:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_7 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:04:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_7 2024-09-15T01:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_7 2024-09-15T01:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-15T01:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-15T01:04:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-15T01:04:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_7 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_7 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-15T01:04:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-15T01:04:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_8 2024-09-15T01:04:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-15T01:04:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:04:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_8 2024-09-15T01:04:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_8 2024-09-15T01:04:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-15T01:04:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-15T01:04:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-15T01:04:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:04:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_8 2024-09-15T01:04:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_8 2024-09-15T01:04:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-15T01:04:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-15T01:04:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_9 2024-09-15T01:04:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_9 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_9 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-15T01:04:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-15T01:04:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_9 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_9 2024-09-15T01:04:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-15T01:04:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-15T01:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_10 2024-09-15T01:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_10 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_10 2024-09-15T01:04:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-15T01:04:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-15T01:04:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_10 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_10 2024-09-15T01:04:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-15T01:04:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-15T01:04:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_11 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_11 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_11 2024-09-15T01:04:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-15T01:04:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-15T01:04:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-15T01:04:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_11 2024-09-15T01:04:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_11 2024-09-15T01:04:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_11 2024-09-15T01:04:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-15T01:04:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-15T01:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_12 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_12 2024-09-15T01:04:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_12 2024-09-15T01:04:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-15T01:04:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-15T01:04:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-15T01:04:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_12 2024-09-15T01:04:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_12 2024-09-15T01:04:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_12 2024-09-15T01:04:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-15T01:04:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-15T01:04:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_13 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_13 2024-09-15T01:04:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_13 2024-09-15T01:04:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-15T01:04:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-15T01:04:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-15T01:04:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_13 2024-09-15T01:04:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_13 2024-09-15T01:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_13 2024-09-15T01:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-15T01:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-15T01:04:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_14 2024-09-15T01:04:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_14 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_14 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-15T01:04:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-15T01:04:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-15T01:04:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_14 2024-09-15T01:04:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_14 2024-09-15T01:04:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_14 2024-09-15T01:04:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-15T01:04:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-15T01:04:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_15 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_15 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_15 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-15T01:04:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-15T01:04:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-15T01:04:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_15 2024-09-15T01:04:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_15 2024-09-15T01:04:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_15 2024-09-15T01:04:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-15T01:04:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-15T01:04:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_16 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_16 2024-09-15T01:04:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_16 2024-09-15T01:04:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-15T01:04:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-15T01:04:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-15T01:04:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_16 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_16 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_16 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-15T01:04:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.083 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7224885918223498) > weight (0.25) 2024-09-15T01:04:38.083 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7089167036909149) > weight (0.25) 2024-09-15T01:04:38.083 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9249966600597594) > weight (0.25) 2024-09-15T01:04:38.083 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.29254688938378093) > weight (0.25) 2024-09-15T01:04:38.083 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2024-09-15T01:04:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-15T01:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_17 2024-09-15T01:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_17 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_17 2024-09-15T01:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-15T01:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-15T01:04:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-15T01:04:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_17 2024-09-15T01:04:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_17 2024-09-15T01:04:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_17 2024-09-15T01:04:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-15T01:04:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-15T01:04:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_18 2024-09-15T01:04:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-15T01:04:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_18 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_18 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-15T01:04:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-15T01:04:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-15T01:04:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_18 2024-09-15T01:04:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_18 2024-09-15T01:04:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_18 2024-09-15T01:04:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-15T01:04:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-15T01:04:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_19 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_19 2024-09-15T01:04:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_19 2024-09-15T01:04:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-15T01:04:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-15T01:04:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_19 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_19 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_19 2024-09-15T01:04:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-15T01:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-15T01:04:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_20 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_20 2024-09-15T01:04:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_20 2024-09-15T01:04:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-15T01:04:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-15T01:04:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-15T01:04:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_20 2024-09-15T01:04:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_20 2024-09-15T01:04:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_20 2024-09-15T01:04:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-15T01:04:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_21 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_21 2024-09-15T01:04:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_21 2024-09-15T01:04:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-15T01:04:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-15T01:04:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_21 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_21 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_21 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-15T01:04:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-15T01:04:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_22 2024-09-15T01:04:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-15T01:04:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_22 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_22 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-15T01:04:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-15T01:04:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-15T01:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_22 2024-09-15T01:04:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_22 2024-09-15T01:04:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_22 2024-09-15T01:04:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-15T01:04:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-15T01:04:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_23 2024-09-15T01:04:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-15T01:04:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T01:04:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_23 2024-09-15T01:04:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_23 2024-09-15T01:04:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-15T01:04:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-15T01:04:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-15T01:04:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_23 2024-09-15T01:04:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_23 2024-09-15T01:04:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_23 2024-09-15T01:04:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-15T01:04:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-15T01:04:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_24 2024-09-15T01:04:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-15T01:04:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T01:04:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_24 2024-09-15T01:04:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_24 2024-09-15T01:04:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-15T01:04:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-15T01:04:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-15T01:04:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_24 2024-09-15T01:04:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_24 2024-09-15T01:04:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_24 2024-09-15T01:04:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-15T01:04:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-15T01:04:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_25 2024-09-15T01:04:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-15T01:04:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_25 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_25 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-15T01:04:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-15T01:04:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_25 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_25 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_25 2024-09-15T01:04:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-15T01:04:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_26 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T01:04:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_26 2024-09-15T01:04:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_26 2024-09-15T01:04:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-15T01:04:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-15T01:04:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-15T01:04:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_26 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_26 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_26 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-15T01:04:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-15T01:04:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_27 2024-09-15T01:04:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-15T01:04:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_27 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_27 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-15T01:04:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-15T01:04:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_27 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_27 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_27 2024-09-15T01:04:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-15T01:04:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-15T01:04:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_28 2024-09-15T01:04:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_28 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_28 2024-09-15T01:04:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-15T01:04:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-15T01:04:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-15T01:04:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_28 2024-09-15T01:04:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_28 2024-09-15T01:04:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_28 2024-09-15T01:04:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-15T01:04:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-15T01:04:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_29 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_29 2024-09-15T01:04:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_29 2024-09-15T01:04:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-15T01:04:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-15T01:04:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-15T01:04:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_29 2024-09-15T01:04:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_29 2024-09-15T01:04:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_29 2024-09-15T01:04:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-15T01:04:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-15T01:04:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_30 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_30 2024-09-15T01:04:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_30 2024-09-15T01:04:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-15T01:04:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-15T01:04:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-15T01:04:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_30 2024-09-15T01:04:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_30 2024-09-15T01:04:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_30 2024-09-15T01:04:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-15T01:04:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-15T01:04:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_31 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_31 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_31 2024-09-15T01:04:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-15T01:04:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-15T01:04:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-15T01:04:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_31 2024-09-15T01:04:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_31 2024-09-15T01:04:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_31 2024-09-15T01:04:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-15T01:04:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-15T01:04:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_32 2024-09-15T01:04:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_32 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_32 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-15T01:04:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-15T01:04:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-15T01:04:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_32 2024-09-15T01:04:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_32 2024-09-15T01:04:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_32 2024-09-15T01:04:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-15T01:04:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-15T01:04:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_33 2024-09-15T01:04:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-15T01:04:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T01:04:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_33 2024-09-15T01:04:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_33 2024-09-15T01:04:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-15T01:04:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-15T01:04:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_33 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_33 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_33 2024-09-15T01:04:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-15T01:04:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-15T01:04:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_34 2024-09-15T01:04:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-15T01:04:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_34 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_34 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-15T01:04:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-15T01:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-15T01:04:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_34 2024-09-15T01:04:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_34 2024-09-15T01:04:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_34 2024-09-15T01:04:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-15T01:04:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-15T01:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_35 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_35 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_35 2024-09-15T01:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-15T01:04:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-15T01:04:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-15T01:04:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_35 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_35 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_35 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-15T01:04:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-15T01:04:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_36 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_36 2024-09-15T01:04:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_36 2024-09-15T01:04:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-15T01:04:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-15T01:04:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-15T01:04:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_36 2024-09-15T01:04:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_36 2024-09-15T01:04:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_36 2024-09-15T01:04:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-15T01:04:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-15T01:04:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_37 2024-09-15T01:04:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_37 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_37 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-15T01:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-15T01:04:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-15T01:04:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_37 2024-09-15T01:04:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_37 2024-09-15T01:04:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_37 2024-09-15T01:04:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-15T01:04:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-15T01:04:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_38 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_38 2024-09-15T01:04:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_38 2024-09-15T01:04:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-15T01:04:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-15T01:04:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_38 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_38 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_38 2024-09-15T01:04:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-15T01:04:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-15T01:04:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_39 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_39 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_39 2024-09-15T01:04:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-15T01:04:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-15T01:04:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-15T01:04:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_39 2024-09-15T01:04:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_39 2024-09-15T01:04:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_39 2024-09-15T01:04:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-15T01:04:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-15T01:04:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_40 2024-09-15T01:04:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-15T01:04:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:04:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_40 2024-09-15T01:04:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_40 2024-09-15T01:04:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-15T01:04:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-15T01:04:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-15T01:04:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_40 2024-09-15T01:04:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_40 2024-09-15T01:04:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_40 2024-09-15T01:04:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-15T01:04:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-15T01:04:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_41 2024-09-15T01:04:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-15T01:04:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:04:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_41 2024-09-15T01:04:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_41 2024-09-15T01:04:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-15T01:04:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-15T01:04:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-15T01:04:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_41 2024-09-15T01:04:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_41 2024-09-15T01:04:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_41 2024-09-15T01:04:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-15T01:04:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-15T01:04:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_42 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_42 2024-09-15T01:04:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_42 2024-09-15T01:04:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-15T01:04:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-15T01:04:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-15T01:04:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_42 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_42 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_42 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-15T01:04:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-15T01:04:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_43 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_43 2024-09-15T01:04:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_43 2024-09-15T01:04:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-15T01:04:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-15T01:04:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-15T01:04:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_43 2024-09-15T01:04:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_43 2024-09-15T01:04:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_43 2024-09-15T01:04:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-15T01:04:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-15T01:04:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_44 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_44 2024-09-15T01:04:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_44 2024-09-15T01:04:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-15T01:04:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-15T01:04:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-15T01:04:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_44 2024-09-15T01:04:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_44 2024-09-15T01:04:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_44 2024-09-15T01:04:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-15T01:04:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-15T01:04:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_45 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_45 2024-09-15T01:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_45 2024-09-15T01:04:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-15T01:04:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-15T01:04:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-15T01:04:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_45 2024-09-15T01:04:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_45 2024-09-15T01:04:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_45 2024-09-15T01:04:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-15T01:04:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-15T01:04:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_46 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_46 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_46 2024-09-15T01:04:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-15T01:04:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-15T01:04:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-15T01:04:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_46 2024-09-15T01:04:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_46 2024-09-15T01:04:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_46 2024-09-15T01:04:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-15T01:04:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-15T01:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_47 2024-09-15T01:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-15T01:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_47 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_47 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-15T01:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-15T01:04:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-15T01:04:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_47 2024-09-15T01:04:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_47 2024-09-15T01:04:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_47 2024-09-15T01:04:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-15T01:04:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-15T01:04:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_48 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_48 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_48 2024-09-15T01:04:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-15T01:04:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-15T01:04:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-15T01:04:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_48 2024-09-15T01:04:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_48 2024-09-15T01:04:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_48 2024-09-15T01:04:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-15T01:04:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-15T01:04:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_49 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_49 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_49 2024-09-15T01:04:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-15T01:04:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-15T01:04:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_49 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_49 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_49 2024-09-15T01:04:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-15T01:04:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-15T01:04:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-15T01:04:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_50 2024-09-15T01:04:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-15T01:04:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_50 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_50 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-15T01:04:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-15T01:04:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:04:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-15T01:04:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_50 2024-09-15T01:04:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:04:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_50 2024-09-15T01:04:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_50 2024-09-15T01:04:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-15T01:04:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-15T01:04:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:04:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:04:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-15T01:04:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ tee -a /tmp/tmp.nS85ynkalX/rbd-mirror.test.log 2024-09-15T01:04:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ date '+%F %T' 2024-09-15T01:04:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo 2024-09-15 01:04:44 TEST: delete images during bootstrap 2024-09-15T01:04:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:2024-09-15 01:04:44 TEST: delete images during bootstrap 2024-09-15T01:04:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-15T01:04:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=image 2024-09-15T01:04:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-15T01:04:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd: mirroring is already configured for image mode 2024-09-15T01:04:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-15T01:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=image 2024-09-15T01:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-15T01:04:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd: mirroring is already configured for image mode 2024-09-15T01:04:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ start_mirror cluster1 2024-09-15T01:04:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local val=cluster1 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster_var_name=cluster 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local instance_var_name=instance 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cluster=cluster1 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ instance=cluster1 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ instance=0 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval cluster=cluster1 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ cluster=cluster1 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ eval instance=0 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ instance=0 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T01:04:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T01:04:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=test 2024-09-15T01:04:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 1 10 2024-09-15T01:04:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T01:04:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_1 2024-09-15T01:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-15T01:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T01:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T01:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T01:04:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:04:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-15T01:04:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-15T01:04:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:04:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T01:04:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-15T01:04:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T01:04:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-15T01:04:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_1': 2024-09-15T01:04:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T01:04:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:04:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T01:04:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1943e2dc0a5 2024-09-15T01:04:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1943e2dc0a5 2024-09-15T01:04:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:04:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T01:04:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:04:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:04:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 01:04:44 2024 2024-09-15T01:04:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 01:04:44 2024 2024-09-15T01:04:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 01:04:44 2024 2024-09-15T01:04:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:04:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:04:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: a64c2b26-84d8-4500-b0a0-5ad3365410ed 2024-09-15T01:04:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T01:04:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-15T01:04:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:04:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T01:04:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=6 2024-09-15T01:04:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:04:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-15T01:04:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 2710528 bytes 2024-09-15T01:04:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 3332096 bytes 2024-09-15T01:04:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 8396800 bytes 2024-09-15T01:04:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 13772800 bytes 2024-09-15T01:04:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 38046720 bytes 2024-09-15T01:04:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 99450880 bytes 2024-09-15T01:04:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 163095552 bytes 2024-09-15T01:04:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 211858432 bytes 2024-09-15T01:04:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 260404224 bytes 2024-09-15T01:04:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 332334080 bytes 2024-09-15T01:04:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 365103104 bytes 2024-09-15T01:04:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 388247552 bytes 2024-09-15T01:04:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 471620608 bytes 2024-09-15T01:04:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 565391360 bytes 2024-09-15T01:04:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 609392640 bytes 2024-09-15T01:04:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 742908928 bytes 2024-09-15T01:04:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 768444416 bytes 2024-09-15T01:04:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 787482624 bytes 2024-09-15T01:04:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 850898944 bytes 2024-09-15T01:04:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 851504128 bytes 2024-09-15T01:04:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 905582592 bytes 2024-09-15T01:04:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 922451968 bytes 2024-09-15T01:04:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 969250816 bytes 2024-09-15T01:04:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 992638976 bytes 2024-09-15T01:04:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 993528832 bytes 2024-09-15T01:04:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1033041920 bytes 2024-09-15T01:04:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1078837248 bytes 2024-09-15T01:04:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1125176320 bytes 2024-09-15T01:04:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1147951104 bytes 2024-09-15T01:04:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1211994112 bytes 2024-09-15T01:04:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1267612672 bytes 2024-09-15T01:04:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1332238336 bytes 2024-09-15T01:04:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1351214080 bytes 2024-09-15T01:04:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1444823040 bytes 2024-09-15T01:04:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1465650176 bytes 2024-09-15T01:04:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1514465280 bytes 2024-09-15T01:04:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1536678912 bytes 2024-09-15T01:04:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1544877056 bytes 2024-09-15T01:04:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1572978688 bytes 2024-09-15T01:04:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1612417024 bytes 2024-09-15T01:04:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1614007296 bytes 2024-09-15T01:04:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1652683776 bytes 2024-09-15T01:04:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1670009856 bytes 2024-09-15T01:04:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1686017024 bytes 2024-09-15T01:04:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1747118080 bytes 2024-09-15T01:04:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1747927040 bytes 2024-09-15T01:04:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1776905216 bytes 2024-09-15T01:04:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1841399808 bytes 2024-09-15T01:04:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1892528128 bytes 2024-09-15T01:04:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T01:04:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:04:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T01:04:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-15T01:04:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=present 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_1 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_1 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-15T01:04:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id=126e5d341e71 2024-09-15T01:04:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:04:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_1 present 126e5d341e71 2024-09-15T01:04:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:04:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:04:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=present 2024-09-15T01:04:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id=126e5d341e71 2024-09-15T01:04:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:04:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:04:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_1 2024-09-15T01:04:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:04:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:04:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_1 2024-09-15T01:04:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-15T01:04:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:04:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id=126e5d341e71 2024-09-15T01:04:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 126e5d341e71 2024-09-15T01:04:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -z 126e5d341e71 -o 126e5d341e71 = 126e5d341e71 2024-09-15T01:04:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=present 2024-09-15T01:04:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test present = present 2024-09-15T01:04:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:04:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-15T01:04:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:04:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:04:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:04:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-15T01:04:56.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:56.291+0000 7fd6b049a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8de4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:04:56.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:04:56.291+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4b900 [3/a64c2b26-84d8-4500-b0a0-5ad3365410ed] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:05:15.878 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3349313523553443) > weight (0.25) 2024-09-15T01:05:15.879 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.36208082563596156) > weight (0.25) 2024-09-15T01:05:15.879 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-15T01:05:15.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-15T01:05:15.880 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-15T01:05:15.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:05:15.879+0000 7f1739c76640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 45191) UID: 1000 2024-09-15T01:05:18.880 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-15T01:05:18.880 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:05:18.881 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-15T01:05:18.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-15T01:05:18.881 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-15T01:05:18.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-15T01:05:18.885 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-15T01:05:19.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:05:19.969+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8de820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:05:19.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:05:19.969+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4b900 [3/a64c2b26-84d8-4500-b0a0-5ad3365410ed] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:05:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring disabled 2024-09-15T01:05:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-15T01:05:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_1 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_1 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-15T01:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:05:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-15T01:05:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:05:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:05:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:05:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_1 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_1 2024-09-15T01:05:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-15T01:05:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:05:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-15T01:05:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:05:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:05:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:05:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:05:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-15T01:05:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:05:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:05:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-15T01:05:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-15T01:05:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:05:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:05:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:05:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:05:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_1 2024-09-15T01:05:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:05:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_1 2024-09-15T01:05:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-15T01:05:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-15T01:05:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_2 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:05:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T01:05:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T01:05:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T01:05:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:05:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-15T01:05:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-15T01:05:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:05:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:05:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T01:05:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-15T01:05:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T01:05:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-15T01:05:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_2': 2024-09-15T01:05:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1961f3cfafa0 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1961f3cfafa0 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 01:05:22 2024 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 01:05:22 2024 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 01:05:22 2024 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:05:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:05:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 23007470-af67-4eb0-87e3-0d07e17ab343 2024-09-15T01:05:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T01:05:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-15T01:05:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:05:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:05:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T01:05:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=11 2024-09-15T01:05:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:05:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-15T01:05:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 34008064 bytes 2024-09-15T01:05:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 71688192 bytes 2024-09-15T01:05:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 99488768 bytes 2024-09-15T01:05:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 108488704 bytes 2024-09-15T01:05:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 133465088 bytes 2024-09-15T01:05:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 179587072 bytes 2024-09-15T01:05:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 201172992 bytes 2024-09-15T01:05:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 208851968 bytes 2024-09-15T01:05:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 224371712 bytes 2024-09-15T01:05:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 224988160 bytes 2024-09-15T01:05:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 269422592 bytes 2024-09-15T01:05:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 319910912 bytes 2024-09-15T01:05:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 386253824 bytes 2024-09-15T01:05:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 403688448 bytes 2024-09-15T01:05:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 448169984 bytes 2024-09-15T01:05:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 449865728 bytes 2024-09-15T01:05:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 477966336 bytes 2024-09-15T01:05:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 499788800 bytes 2024-09-15T01:05:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 501272576 bytes 2024-09-15T01:05:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 530654208 bytes 2024-09-15T01:05:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 570818560 bytes 2024-09-15T01:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 601455616 bytes 2024-09-15T01:05:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 687415296 bytes 2024-09-15T01:05:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 722134016 bytes 2024-09-15T01:05:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 777658368 bytes 2024-09-15T01:05:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 858019840 bytes 2024-09-15T01:05:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 926089216 bytes 2024-09-15T01:05:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 980971520 bytes 2024-09-15T01:05:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1036440576 bytes 2024-09-15T01:05:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1039108096 bytes 2024-09-15T01:05:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1077062656 bytes 2024-09-15T01:05:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1077682176 bytes 2024-09-15T01:05:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1142056960 bytes 2024-09-15T01:05:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1199973376 bytes 2024-09-15T01:05:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1287812096 bytes 2024-09-15T01:05:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1288425472 bytes 2024-09-15T01:05:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1327510528 bytes 2024-09-15T01:05:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1353156608 bytes 2024-09-15T01:05:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1469114368 bytes 2024-09-15T01:05:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1538318336 bytes 2024-09-15T01:05:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1596616704 bytes 2024-09-15T01:05:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1681881088 bytes 2024-09-15T01:05:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1701939200 bytes 2024-09-15T01:05:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1723677696 bytes 2024-09-15T01:05:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1742302208 bytes 2024-09-15T01:05:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1779010560 bytes 2024-09-15T01:05:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1779660800 bytes 2024-09-15T01:05:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1797295104 bytes 2024-09-15T01:05:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1871559680 bytes 2024-09-15T01:05:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1915392000 bytes 2024-09-15T01:05:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2002630656 bytes 2024-09-15T01:05:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2058305536 bytes 2024-09-15T01:05:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2058955776 bytes 2024-09-15T01:05:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2079577088 bytes 2024-09-15T01:05:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2126029824 bytes 2024-09-15T01:05:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2188108800 bytes 2024-09-15T01:05:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2240128000 bytes 2024-09-15T01:05:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2266655744 bytes 2024-09-15T01:05:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2367816704 bytes 2024-09-15T01:05:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2414535680 bytes 2024-09-15T01:05:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2491021312 bytes 2024-09-15T01:05:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2509075456 bytes 2024-09-15T01:05:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2564861952 bytes 2024-09-15T01:05:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2599840768 bytes 2024-09-15T01:05:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2627912704 bytes 2024-09-15T01:05:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2628535296 bytes 2024-09-15T01:05:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2670925824 bytes 2024-09-15T01:05:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2713521152 bytes 2024-09-15T01:05:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2735478784 bytes 2024-09-15T01:05:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2773295104 bytes 2024-09-15T01:05:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2820669440 bytes 2024-09-15T01:05:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2859475968 bytes 2024-09-15T01:05:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2906372096 bytes 2024-09-15T01:05:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2935061504 bytes 2024-09-15T01:05:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2986299392 bytes 2024-09-15T01:05:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3061329920 bytes 2024-09-15T01:05:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3119172608 bytes 2024-09-15T01:05:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3161744384 bytes 2024-09-15T01:05:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3226768384 bytes 2024-09-15T01:05:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3232322560 bytes 2024-09-15T01:05:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3288846336 bytes 2024-09-15T01:05:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3297277952 bytes 2024-09-15T01:05:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3376608256 bytes 2024-09-15T01:05:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3377914880 bytes 2024-09-15T01:05:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3381678080 bytes 2024-09-15T01:05:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3409043456 bytes 2024-09-15T01:05:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3428730880 bytes 2024-09-15T01:05:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3462688768 bytes 2024-09-15T01:05:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3567922176 bytes 2024-09-15T01:05:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3604311040 bytes 2024-09-15T01:05:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3667033088 bytes 2024-09-15T01:05:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T01:05:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:05:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T01:05:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:05:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-15T01:05:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:05:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=present 2024-09-15T01:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_2 2024-09-15T01:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_2 2024-09-15T01:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-15T01:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:05:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id=1625450210e2 2024-09-15T01:05:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:05:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:05:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_2 present 1625450210e2 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=present 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id=1625450210e2 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_2 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_2 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-15T01:05:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:05:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id=1625450210e2 2024-09-15T01:05:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1625450210e2 2024-09-15T01:05:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -z 1625450210e2 -o 1625450210e2 = 1625450210e2 2024-09-15T01:05:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=present 2024-09-15T01:05:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test present = present 2024-09-15T01:05:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:05:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-15T01:05:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:05:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:05:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:05:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-15T01:05:56.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:05:56.290+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9c8e0ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:05:56.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:05:56.290+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9d3d0780 [3/23007470-af67-4eb0-87e3-0d07e17ab343] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:06:04.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:06:04.962+0000 7f6a19abd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d67646ac00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2024-09-15T01:06:04.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:06:04.962+0000 7f6a19abd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d67646ac00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=186, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=1108ad63-b7b4-4889-ae31-b1fd6de8431b, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2024-09-15T01:06:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring disabled 2024-09-15T01:06:04.975 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:06:04.973+0000 7fc49cacc640 -1 reset not still connected to 0x560c68a39380 2024-09-15T01:06:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-15T01:06:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:06:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:06:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:06:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:06:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:06:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:06:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_2 2024-09-15T01:06:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:06:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:06:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_2 2024-09-15T01:06:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:06:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-15T01:06:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-15T01:06:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:06:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:06:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-15T01:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:06:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_2 2024-09-15T01:06:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:06:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:06:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_2 2024-09-15T01:06:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-15T01:06:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:06:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-15T01:06:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:06:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:06:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:06:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:06:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-15T01:06:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:06:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-15T01:06:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-15T01:06:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:06:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:06:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:06:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_2 2024-09-15T01:06:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_2 2024-09-15T01:06:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-15T01:06:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-15T01:06:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T01:06:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:06:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T01:06:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_3 2024-09-15T01:06:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-15T01:06:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:06:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:06:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:06:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T01:06:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T01:06:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T01:06:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:06:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-15T01:06:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-15T01:06:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T01:06:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-15T01:06:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T01:06:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-15T01:06:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_3': 2024-09-15T01:06:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T01:06:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:06:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T01:06:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 197958e1c708 2024-09-15T01:06:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.197958e1c708 2024-09-15T01:06:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:06:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T01:06:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:06:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:06:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 01:06:05 2024 2024-09-15T01:06:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 01:06:05 2024 2024-09-15T01:06:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 01:06:05 2024 2024-09-15T01:06:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:06:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:06:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 670a434a-8ff0-41e2-b32a-a5bf141b17ab 2024-09-15T01:06:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T01:06:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-15T01:06:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T01:06:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=14 2024-09-15T01:06:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:06:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-15T01:06:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 64520192 bytes 2024-09-15T01:06:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 129744896 bytes 2024-09-15T01:06:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 195959808 bytes 2024-09-15T01:06:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 255983616 bytes 2024-09-15T01:06:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 291986432 bytes 2024-09-15T01:06:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 328596480 bytes 2024-09-15T01:06:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 349267968 bytes 2024-09-15T01:06:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 370556928 bytes 2024-09-15T01:06:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 409871360 bytes 2024-09-15T01:06:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 422680576 bytes 2024-09-15T01:06:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 517590016 bytes 2024-09-15T01:06:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 549234688 bytes 2024-09-15T01:06:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 586011648 bytes 2024-09-15T01:06:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 635670528 bytes 2024-09-15T01:06:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 704756736 bytes 2024-09-15T01:06:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 735351808 bytes 2024-09-15T01:06:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 824990720 bytes 2024-09-15T01:06:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 862175232 bytes 2024-09-15T01:06:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 975281152 bytes 2024-09-15T01:06:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1008734208 bytes 2024-09-15T01:06:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1107504128 bytes 2024-09-15T01:06:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1135188992 bytes 2024-09-15T01:06:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1275640832 bytes 2024-09-15T01:06:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1312755712 bytes 2024-09-15T01:06:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1342747648 bytes 2024-09-15T01:06:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1395379200 bytes 2024-09-15T01:06:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1421973504 bytes 2024-09-15T01:06:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1454514176 bytes 2024-09-15T01:06:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1499542528 bytes 2024-09-15T01:06:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1500171264 bytes 2024-09-15T01:06:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1589493760 bytes 2024-09-15T01:06:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1628716032 bytes 2024-09-15T01:06:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1680927744 bytes 2024-09-15T01:06:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1713495040 bytes 2024-09-15T01:06:10.664 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1789204480 bytes 2024-09-15T01:06:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1809317888 bytes 2024-09-15T01:06:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1833249792 bytes 2024-09-15T01:06:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1857214464 bytes 2024-09-15T01:06:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1905431552 bytes 2024-09-15T01:06:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1926761472 bytes 2024-09-15T01:06:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1942478848 bytes 2024-09-15T01:06:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1963120640 bytes 2024-09-15T01:06:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2014369792 bytes 2024-09-15T01:06:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2067720192 bytes 2024-09-15T01:06:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2106863616 bytes 2024-09-15T01:06:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2107472896 bytes 2024-09-15T01:06:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2132849664 bytes 2024-09-15T01:06:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2194095104 bytes 2024-09-15T01:06:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2276441088 bytes 2024-09-15T01:06:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2307235840 bytes 2024-09-15T01:06:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2347197440 bytes 2024-09-15T01:06:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2393666560 bytes 2024-09-15T01:06:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2445300736 bytes 2024-09-15T01:06:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2481466368 bytes 2024-09-15T01:06:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2521522176 bytes 2024-09-15T01:06:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2534549504 bytes 2024-09-15T01:06:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2568431616 bytes 2024-09-15T01:06:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2640070656 bytes 2024-09-15T01:06:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2716019712 bytes 2024-09-15T01:06:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2816546816 bytes 2024-09-15T01:06:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2840580096 bytes 2024-09-15T01:06:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2885556224 bytes 2024-09-15T01:06:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2910337024 bytes 2024-09-15T01:06:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2944491520 bytes 2024-09-15T01:06:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 3036570624 bytes 2024-09-15T01:06:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 3049334784 bytes 2024-09-15T01:06:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3117960192 bytes 2024-09-15T01:06:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3151383552 bytes 2024-09-15T01:06:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3176161280 bytes 2024-09-15T01:06:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3224770560 bytes 2024-09-15T01:06:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3290929152 bytes 2024-09-15T01:06:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3354236928 bytes 2024-09-15T01:06:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3411083264 bytes 2024-09-15T01:06:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3447634944 bytes 2024-09-15T01:06:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3452903424 bytes 2024-09-15T01:06:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3479600128 bytes 2024-09-15T01:06:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3508783104 bytes 2024-09-15T01:06:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3525713920 bytes 2024-09-15T01:06:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3553793024 bytes 2024-09-15T01:06:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3594339328 bytes 2024-09-15T01:06:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3654217728 bytes 2024-09-15T01:06:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3705596928 bytes 2024-09-15T01:06:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3716120576 bytes 2024-09-15T01:06:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3761658880 bytes 2024-09-15T01:06:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3832473600 bytes 2024-09-15T01:06:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3868108800 bytes 2024-09-15T01:06:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3915602944 bytes 2024-09-15T01:06:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3951480832 bytes 2024-09-15T01:06:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3952340992 bytes 2024-09-15T01:06:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3958702080 bytes 2024-09-15T01:06:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3959325696 bytes 2024-09-15T01:06:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 4043820032 bytes 2024-09-15T01:06:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 4074502144 bytes 2024-09-15T01:06:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 4109966336 bytes 2024-09-15T01:06:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4110576640 bytes 2024-09-15T01:06:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4135119872 bytes 2024-09-15T01:06:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4188565504 bytes 2024-09-15T01:06:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4230478848 bytes 2024-09-15T01:06:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4278650880 bytes 2024-09-15T01:06:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4305506304 bytes 2024-09-15T01:06:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4333803520 bytes 2024-09-15T01:06:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4402908160 bytes 2024-09-15T01:06:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4456502272 bytes 2024-09-15T01:06:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10400 IOs, 4491935744 bytes 2024-09-15T01:06:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10500 IOs, 4514091008 bytes 2024-09-15T01:06:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10600 IOs, 4536083456 bytes 2024-09-15T01:06:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10700 IOs, 4587621376 bytes 2024-09-15T01:06:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10800 IOs, 4684537856 bytes 2024-09-15T01:06:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10900 IOs, 4714130432 bytes 2024-09-15T01:06:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11000 IOs, 4738965504 bytes 2024-09-15T01:06:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11100 IOs, 4784882688 bytes 2024-09-15T01:06:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11200 IOs, 4825703424 bytes 2024-09-15T01:06:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:11300 IOs, 4900893696 bytes 2024-09-15T01:06:20.032 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=present 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_3 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:06:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:06:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_3 2024-09-15T01:06:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-15T01:06:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:06:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id=16253c08cc22 2024-09-15T01:06:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:06:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:06:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_3 present 16253c08cc22 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=present 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id=16253c08cc22 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_3 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_3 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-15T01:06:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:06:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id=16253c08cc22 2024-09-15T01:06:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 16253c08cc22 2024-09-15T01:06:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -z 16253c08cc22 -o 16253c08cc22 = 16253c08cc22 2024-09-15T01:06:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=present 2024-09-15T01:06:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test present = present 2024-09-15T01:06:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:06:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-15T01:06:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-15T01:06:26.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:06:26.290+0000 7fd6b0c9b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cf9cf0fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:06:26.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:06:26.290+0000 7fd6b6ca7640 -1 rbd::mirror::ImageReplayer: 0x55cf9ce4b900 [3/670a434a-8ff0-41e2-b32a-a5bf141b17ab] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:06:30.084 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-15T01:06:30.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-15T01:06:30.085 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-15T01:06:30.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:06:30.085+0000 7fd6b7ca9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 49009) UID: 1000 2024-09-15T01:06:31.088 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-15T01:06:31.088 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:06:31.088 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-15T01:06:31.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-15T01:06:31.089 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-15T01:06:31.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-15T01:06:31.092 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-09-15T01:06:34.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:06:34.143+0000 7fc726c04640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561920c65860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:06:34.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:06:34.143+0000 7fc72d411640 -1 rbd::mirror::ImageReplayer: 0x561920bb2280 [3/670a434a-8ff0-41e2-b32a-a5bf141b17ab] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:06:51.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:06:51.906+0000 7f6a1a2be640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d6762c9800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2024-09-15T01:06:51.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:06:51.906+0000 7f6a1a2be640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d6762c9800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=187, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=1108ad63-b7b4-4889-ae31-b1fd6de8431b, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2024-09-15T01:06:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring disabled 2024-09-15T01:06:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-15T01:06:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_3 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_3 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-15T01:06:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:06:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-15T01:06:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:06:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:06:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:06:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-15T01:06:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_3 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_3 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-15T01:06:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:06:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-15T01:06:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:06:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:06:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:06:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:06:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-15T01:06:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-15T01:06:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-15T01:06:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:06:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:06:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_3 2024-09-15T01:06:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_3 2024-09-15T01:06:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-15T01:06:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-15T01:06:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T01:06:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:06:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T01:06:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_4 2024-09-15T01:06:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-15T01:06:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:06:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:06:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:06:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:06:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T01:06:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T01:06:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T01:06:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:06:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-15T01:06:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-15T01:06:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:06:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T01:06:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-15T01:06:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T01:06:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-15T01:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_4': 2024-09-15T01:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T01:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T01:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 199ac9b1636a 2024-09-15T01:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.199ac9b1636a 2024-09-15T01:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T01:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:06:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:06:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 01:06:52 2024 2024-09-15T01:06:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 01:06:52 2024 2024-09-15T01:06:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 01:06:52 2024 2024-09-15T01:06:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:06:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:06:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 547a74d0-6307-4041-9f6c-ee577d63e42a 2024-09-15T01:06:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T01:06:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-15T01:06:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:06:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:06:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:06:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T01:06:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=8 2024-09-15T01:06:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:06:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-15T01:06:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 6937600 bytes 2024-09-15T01:06:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 50775040 bytes 2024-09-15T01:06:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 120517632 bytes 2024-09-15T01:06:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 190539776 bytes 2024-09-15T01:06:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 234446848 bytes 2024-09-15T01:06:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 298462208 bytes 2024-09-15T01:06:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 378918912 bytes 2024-09-15T01:06:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 408639488 bytes 2024-09-15T01:06:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 431623168 bytes 2024-09-15T01:06:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 459893760 bytes 2024-09-15T01:06:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 503782400 bytes 2024-09-15T01:06:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 566305792 bytes 2024-09-15T01:06:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 601342976 bytes 2024-09-15T01:06:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 630564864 bytes 2024-09-15T01:06:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 640766976 bytes 2024-09-15T01:06:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 698231808 bytes 2024-09-15T01:06:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 749802496 bytes 2024-09-15T01:06:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 757917696 bytes 2024-09-15T01:06:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 775048192 bytes 2024-09-15T01:06:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 849088512 bytes 2024-09-15T01:06:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 922269696 bytes 2024-09-15T01:06:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 946752512 bytes 2024-09-15T01:06:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1039978496 bytes 2024-09-15T01:06:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1074724864 bytes 2024-09-15T01:06:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1117780992 bytes 2024-09-15T01:06:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1137072128 bytes 2024-09-15T01:06:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1212312576 bytes 2024-09-15T01:06:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1263401984 bytes 2024-09-15T01:06:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1264015360 bytes 2024-09-15T01:06:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1281061888 bytes 2024-09-15T01:06:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1320109056 bytes 2024-09-15T01:06:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1320719360 bytes 2024-09-15T01:06:57.091 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1370236928 bytes 2024-09-15T01:06:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1370867712 bytes 2024-09-15T01:06:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1376280576 bytes 2024-09-15T01:06:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1412983808 bytes 2024-09-15T01:06:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1491081216 bytes 2024-09-15T01:06:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1535452160 bytes 2024-09-15T01:06:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1571168256 bytes 2024-09-15T01:06:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1589023744 bytes 2024-09-15T01:06:57.886 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6267248337508508) > weight (0.25) 2024-09-15T01:06:57.886 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-15T01:06:57.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-15T01:06:57.886 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-15T01:06:57.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:06:57.886+0000 7f29e098f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 61833) UID: 1000 2024-09-15T01:06:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1637394432 bytes 2024-09-15T01:06:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1666568192 bytes 2024-09-15T01:06:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1702250496 bytes 2024-09-15T01:06:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1743796224 bytes 2024-09-15T01:06:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1832895488 bytes 2024-09-15T01:06:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1874028544 bytes 2024-09-15T01:06:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1932239872 bytes 2024-09-15T01:06:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1943814144 bytes 2024-09-15T01:06:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1984501760 bytes 2024-09-15T01:06:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2011919360 bytes 2024-09-15T01:06:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2057737216 bytes 2024-09-15T01:06:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2082831360 bytes 2024-09-15T01:06:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2095270912 bytes 2024-09-15T01:06:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2111613952 bytes 2024-09-15T01:06:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2151933952 bytes 2024-09-15T01:06:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2245352448 bytes 2024-09-15T01:06:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2266019840 bytes 2024-09-15T01:07:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2293213184 bytes 2024-09-15T01:07:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2314641408 bytes 2024-09-15T01:07:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2327651328 bytes 2024-09-15T01:07:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2368996352 bytes 2024-09-15T01:07:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2425018368 bytes 2024-09-15T01:07:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2455734272 bytes 2024-09-15T01:07:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2473062400 bytes 2024-09-15T01:07:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2501803008 bytes 2024-09-15T01:07:00.887 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-15T01:07:00.887 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:07:00.888 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-15T01:07:00.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-15T01:07:00.888 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-15T01:07:00.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-15T01:07:00.891 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-09-15T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-15T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=present 2024-09-15T01:07:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:07:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:07:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:07:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_4 2024-09-15T01:07:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:07:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:07:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_4 2024-09-15T01:07:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-15T01:07:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:07:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id=1625c6229484 2024-09-15T01:07:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:07:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_4 present 1625c6229484 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=present 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id=1625c6229484 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_4 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:07:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:07:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_4 2024-09-15T01:07:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-15T01:07:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:07:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id=1625c6229484 2024-09-15T01:07:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1625c6229484 2024-09-15T01:07:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -z 1625c6229484 -o 1625c6229484 = 1625c6229484 2024-09-15T01:07:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=present 2024-09-15T01:07:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test present = present 2024-09-15T01:07:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:07:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-15T01:07:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:07:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:07:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-15T01:07:27.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:07:27.200+0000 7fc726403640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561920c65380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:07:27.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:07:27.200+0000 7fc72d411640 -1 rbd::mirror::ImageReplayer: 0x56192170e780 [3/547a74d0-6307-4041-9f6c-ee577d63e42a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:07:29.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:07:29.689+0000 7f6a19abd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d67649a800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=188 2024-09-15T01:07:29.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:07:29.689+0000 7f6a19abd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d67649a800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=188, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=1108ad63-b7b4-4889-ae31-b1fd6de8431b, primary_snap_id=bc, last_copied_object_number=128, snap_seqs={188=18446744073709551614}] 2024-09-15T01:07:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring disabled 2024-09-15T01:07:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-15T01:07:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:07:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_4 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_4 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-15T01:07:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:07:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-15T01:07:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:07:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:07:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:07:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-15T01:07:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_4 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_4 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-15T01:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:07:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-15T01:07:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:07:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:07:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:07:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:07:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-15T01:07:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:07:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:07:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-15T01:07:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-15T01:07:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:07:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:07:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:07:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:07:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_4 2024-09-15T01:07:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:07:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_4 2024-09-15T01:07:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-15T01:07:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-15T01:07:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T01:07:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:07:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T01:07:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_5 2024-09-15T01:07:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-15T01:07:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:07:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:07:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:07:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:07:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:07:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T01:07:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T01:07:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T01:07:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:07:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-15T01:07:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-15T01:07:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:07:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:07:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T01:07:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-15T01:07:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T01:07:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-15T01:07:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_5': 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 19b8c673e3c1 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.19b8c673e3c1 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 01:07:30 2024 2024-09-15T01:07:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 01:07:30 2024 2024-09-15T01:07:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 01:07:30 2024 2024-09-15T01:07:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:07:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:07:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 6af214aa-8341-4e18-85a0-4544d240b7fa 2024-09-15T01:07:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T01:07:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-15T01:07:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:07:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:07:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T01:07:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=13 2024-09-15T01:07:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:07:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-15T01:07:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 41281536 bytes 2024-09-15T01:07:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 83600384 bytes 2024-09-15T01:07:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 85766144 bytes 2024-09-15T01:07:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 140488704 bytes 2024-09-15T01:07:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 180887552 bytes 2024-09-15T01:07:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 224788480 bytes 2024-09-15T01:07:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 281272320 bytes 2024-09-15T01:07:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 309321728 bytes 2024-09-15T01:07:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 334769152 bytes 2024-09-15T01:07:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 377887744 bytes 2024-09-15T01:07:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 407822336 bytes 2024-09-15T01:07:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 456593408 bytes 2024-09-15T01:07:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 487814144 bytes 2024-09-15T01:07:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 513770496 bytes 2024-09-15T01:07:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 552695808 bytes 2024-09-15T01:07:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 571077632 bytes 2024-09-15T01:07:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 582923264 bytes 2024-09-15T01:07:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 632315904 bytes 2024-09-15T01:07:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 679832576 bytes 2024-09-15T01:07:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 698437632 bytes 2024-09-15T01:07:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 739549184 bytes 2024-09-15T01:07:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 822365184 bytes 2024-09-15T01:07:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 838863872 bytes 2024-09-15T01:07:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 909223936 bytes 2024-09-15T01:07:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 996468736 bytes 2024-09-15T01:07:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1111377920 bytes 2024-09-15T01:07:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1144895488 bytes 2024-09-15T01:07:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1204153344 bytes 2024-09-15T01:07:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1221798912 bytes 2024-09-15T01:07:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1284329472 bytes 2024-09-15T01:07:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1308208128 bytes 2024-09-15T01:07:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1353671680 bytes 2024-09-15T01:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1389093888 bytes 2024-09-15T01:07:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1426500608 bytes 2024-09-15T01:07:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1448964096 bytes 2024-09-15T01:07:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1504837632 bytes 2024-09-15T01:07:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1541327872 bytes 2024-09-15T01:07:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1626324992 bytes 2024-09-15T01:07:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1717425152 bytes 2024-09-15T01:07:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1737742336 bytes 2024-09-15T01:07:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1790567424 bytes 2024-09-15T01:07:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1791208448 bytes 2024-09-15T01:07:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1846958080 bytes 2024-09-15T01:07:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1925959680 bytes 2024-09-15T01:07:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2030708736 bytes 2024-09-15T01:07:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2035643392 bytes 2024-09-15T01:07:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2052480000 bytes 2024-09-15T01:07:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2122384384 bytes 2024-09-15T01:07:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2155486208 bytes 2024-09-15T01:07:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2194839552 bytes 2024-09-15T01:07:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2230761472 bytes 2024-09-15T01:07:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2270096384 bytes 2024-09-15T01:07:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2347709440 bytes 2024-09-15T01:07:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2407314432 bytes 2024-09-15T01:07:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2416837632 bytes 2024-09-15T01:07:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2491073536 bytes 2024-09-15T01:07:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2566987776 bytes 2024-09-15T01:07:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2576807936 bytes 2024-09-15T01:07:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2659620864 bytes 2024-09-15T01:07:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2735314944 bytes 2024-09-15T01:07:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2735934464 bytes 2024-09-15T01:07:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2763973632 bytes 2024-09-15T01:07:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2794062848 bytes 2024-09-15T01:07:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2814726144 bytes 2024-09-15T01:07:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2891331584 bytes 2024-09-15T01:07:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2912347136 bytes 2024-09-15T01:07:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2918973440 bytes 2024-09-15T01:07:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2944715776 bytes 2024-09-15T01:07:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2973334528 bytes 2024-09-15T01:07:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3015966720 bytes 2024-09-15T01:07:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3056450560 bytes 2024-09-15T01:07:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3109118976 bytes 2024-09-15T01:07:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3130250240 bytes 2024-09-15T01:07:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3208832000 bytes 2024-09-15T01:07:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3224798208 bytes 2024-09-15T01:07:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3250529280 bytes 2024-09-15T01:07:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3287966720 bytes 2024-09-15T01:07:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3320095744 bytes 2024-09-15T01:07:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3384862720 bytes 2024-09-15T01:07:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3466526720 bytes 2024-09-15T01:07:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3467155456 bytes 2024-09-15T01:07:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3553707008 bytes 2024-09-15T01:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3597637632 bytes 2024-09-15T01:07:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3622657024 bytes 2024-09-15T01:07:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3662611456 bytes 2024-09-15T01:07:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3727735808 bytes 2024-09-15T01:07:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3751630848 bytes 2024-09-15T01:07:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3778744320 bytes 2024-09-15T01:07:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3779379200 bytes 2024-09-15T01:07:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3792547840 bytes 2024-09-15T01:07:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3830921216 bytes 2024-09-15T01:07:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3849433088 bytes 2024-09-15T01:07:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3929023488 bytes 2024-09-15T01:07:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3964168192 bytes 2024-09-15T01:07:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 4032588800 bytes 2024-09-15T01:07:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 4113131520 bytes 2024-09-15T01:07:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 4201332736 bytes 2024-09-15T01:07:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9800 IOs, 4235922432 bytes 2024-09-15T01:07:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9900 IOs, 4313061376 bytes 2024-09-15T01:07:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10000 IOs, 4378737664 bytes 2024-09-15T01:07:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10100 IOs, 4451440640 bytes 2024-09-15T01:07:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10200 IOs, 4454990848 bytes 2024-09-15T01:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:10300 IOs, 4541444096 bytes 2024-09-15T01:07:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T01:07:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:07:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=present 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_5 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:07:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_5 2024-09-15T01:07:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-15T01:07:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:07:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id=162548081b92 2024-09-15T01:07:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:07:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:07:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_5 present 162548081b92 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=present 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id=162548081b92 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_5 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:07:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_5 2024-09-15T01:07:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-15T01:07:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id=162548081b92 2024-09-15T01:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 162548081b92 2024-09-15T01:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -z 162548081b92 -o 162548081b92 = 162548081b92 2024-09-15T01:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=present 2024-09-15T01:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test present = present 2024-09-15T01:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-15T01:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:07:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-15T01:07:57.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:07:57.199+0000 7fc726c04640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561920c69520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:07:57.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:07:57.199+0000 7fc72cc10640 -1 rbd::mirror::ImageReplayer: 0x561920bb2280 [3/6af214aa-8341-4e18-85a0-4544d240b7fa] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:08:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring disabled 2024-09-15T01:08:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-15T01:08:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_5 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_5 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-15T01:08:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:08:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-15T01:08:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:08:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:08:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:08:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-15T01:08:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:08:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:08:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:08:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:08:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:08:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:08:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_5 2024-09-15T01:08:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:08:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:08:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_5 2024-09-15T01:08:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-15T01:08:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:08:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-15T01:08:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:08:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:08:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:08:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:08:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-15T01:08:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:08:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-15T01:08:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-15T01:08:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:08:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:08:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:08:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_5 2024-09-15T01:08:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_5 2024-09-15T01:08:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-15T01:08:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-15T01:08:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T01:08:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:08:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T01:08:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_6 2024-09-15T01:08:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-15T01:08:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:08:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:08:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:08:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T01:08:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T01:08:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T01:08:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:08:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-15T01:08:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-15T01:08:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T01:08:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-15T01:08:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T01:08:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-15T01:08:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_6': 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 19d06aea768c 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.19d06aea768c 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 01:08:14 2024 2024-09-15T01:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 01:08:14 2024 2024-09-15T01:08:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 01:08:14 2024 2024-09-15T01:08:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:08:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:08:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 6fd742c8-18a7-4c3a-be3a-b7289bc501f3 2024-09-15T01:08:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T01:08:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-15T01:08:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T01:08:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=11 2024-09-15T01:08:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:08:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-15T01:08:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 14803968 bytes 2024-09-15T01:08:15.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 115964928 bytes 2024-09-15T01:08:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 132622336 bytes 2024-09-15T01:08:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 189914112 bytes 2024-09-15T01:08:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 199950336 bytes 2024-09-15T01:08:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 225188864 bytes 2024-09-15T01:08:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 232040448 bytes 2024-09-15T01:08:16.094 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4338950141366533) > weight (0.25) 2024-09-15T01:08:16.094 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.860474965074749) > weight (0.25) 2024-09-15T01:08:16.094 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-15T01:08:16.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-15T01:08:16.095 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-15T01:08:16.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:08:16.094+0000 7fc72dc12640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 63644) UID: 1000 2024-09-15T01:08:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 298849280 bytes 2024-09-15T01:08:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 309928960 bytes 2024-09-15T01:08:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 398949376 bytes 2024-09-15T01:08:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 407298048 bytes 2024-09-15T01:08:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 451381248 bytes 2024-09-15T01:08:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 510352384 bytes 2024-09-15T01:08:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 583619584 bytes 2024-09-15T01:08:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 625624064 bytes 2024-09-15T01:08:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 702185472 bytes 2024-09-15T01:08:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 784997376 bytes 2024-09-15T01:08:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 805067776 bytes 2024-09-15T01:08:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 877271040 bytes 2024-09-15T01:08:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 984509440 bytes 2024-09-15T01:08:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1004836864 bytes 2024-09-15T01:08:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1056541696 bytes 2024-09-15T01:08:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1091579904 bytes 2024-09-15T01:08:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1197382656 bytes 2024-09-15T01:08:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1259411456 bytes 2024-09-15T01:08:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1325160448 bytes 2024-09-15T01:08:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1358668800 bytes 2024-09-15T01:08:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1391509504 bytes 2024-09-15T01:08:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1447964672 bytes 2024-09-15T01:08:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1535133696 bytes 2024-09-15T01:08:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1546676224 bytes 2024-09-15T01:08:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1550493696 bytes 2024-09-15T01:08:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1577158656 bytes 2024-09-15T01:08:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1624355840 bytes 2024-09-15T01:08:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1662472192 bytes 2024-09-15T01:08:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1697217536 bytes 2024-09-15T01:08:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1765842944 bytes 2024-09-15T01:08:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1777288192 bytes 2024-09-15T01:08:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1823932416 bytes 2024-09-15T01:08:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1851969536 bytes 2024-09-15T01:08:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1925906432 bytes 2024-09-15T01:08:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1949641728 bytes 2024-09-15T01:08:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1950257152 bytes 2024-09-15T01:08:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1984020480 bytes 2024-09-15T01:08:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2040659968 bytes 2024-09-15T01:08:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2087230464 bytes 2024-09-15T01:08:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2108271616 bytes 2024-09-15T01:08:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2161924096 bytes 2024-09-15T01:08:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2228699136 bytes 2024-09-15T01:08:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2242799616 bytes 2024-09-15T01:08:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2326047744 bytes 2024-09-15T01:08:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2326634496 bytes 2024-09-15T01:08:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2382668800 bytes 2024-09-15T01:08:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2393272320 bytes 2024-09-15T01:08:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2443414528 bytes 2024-09-15T01:08:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2516887552 bytes 2024-09-15T01:08:22.095 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-15T01:08:22.095 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:08:22.096 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-15T01:08:22.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-15T01:08:22.096 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-15T01:08:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-15T01:08:22.099 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-15T01:08:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2573404160 bytes 2024-09-15T01:08:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2666174464 bytes 2024-09-15T01:08:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2805096448 bytes 2024-09-15T01:08:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2827799552 bytes 2024-09-15T01:08:22.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2863960064 bytes 2024-09-15T01:08:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2903275520 bytes 2024-09-15T01:08:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2913215488 bytes 2024-09-15T01:08:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2943042560 bytes 2024-09-15T01:08:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2943629312 bytes 2024-09-15T01:08:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 3058349056 bytes 2024-09-15T01:08:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3080180736 bytes 2024-09-15T01:08:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3120474112 bytes 2024-09-15T01:08:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3121099776 bytes 2024-09-15T01:08:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3171736576 bytes 2024-09-15T01:08:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3259537408 bytes 2024-09-15T01:08:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3319870464 bytes 2024-09-15T01:08:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3360398336 bytes 2024-09-15T01:08:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3389838336 bytes 2024-09-15T01:08:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3426930688 bytes 2024-09-15T01:08:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3448322048 bytes 2024-09-15T01:08:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3507863552 bytes 2024-09-15T01:08:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3542505472 bytes 2024-09-15T01:08:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3563291648 bytes 2024-09-15T01:08:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3625334784 bytes 2024-09-15T01:08:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3645897728 bytes 2024-09-15T01:08:25.260 INFO:tasks.ceph.cluster2.osd.2.smithi059.stderr:2024-09-15T01:08:25.259+0000 7fc49cacc640 -1 reset not still connected to 0x560c60f4cea0 2024-09-15T01:08:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3668110336 bytes 2024-09-15T01:08:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3699528704 bytes 2024-09-15T01:08:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3758945280 bytes 2024-09-15T01:08:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3763521536 bytes 2024-09-15T01:08:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3782148096 bytes 2024-09-15T01:08:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3834638336 bytes 2024-09-15T01:08:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T01:08:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:08:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T01:08:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:08:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-15T01:08:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:08:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=present 2024-09-15T01:08:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:08:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_6 2024-09-15T01:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_6 2024-09-15T01:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-15T01:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:08:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id=1625b62867c 2024-09-15T01:08:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:08:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:08:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_6 present 1625b62867c 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=present 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id=1625b62867c 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_6 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_6 2024-09-15T01:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-15T01:08:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:08:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id=1625b62867c 2024-09-15T01:08:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1625b62867c 2024-09-15T01:08:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -z 1625b62867c -o 1625b62867c = 1625b62867c 2024-09-15T01:08:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=present 2024-09-15T01:08:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test present = present 2024-09-15T01:08:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:08:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-15T01:08:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-15T01:08:42.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:08:42.241+0000 7f0d7d103640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ca8019f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:08:42.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:08:42.241+0000 7f0d8310f640 -1 rbd::mirror::ImageReplayer: 0x55ca80ef4280 [3/6fd742c8-18a7-4c3a-be3a-b7289bc501f3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:08:50.893 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9205124000391125) > weight (0.25) 2024-09-15T01:08:50.893 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6728455333979374) > weight (0.25) 2024-09-15T01:08:50.893 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7361956114040067) > weight (0.25) 2024-09-15T01:08:50.893 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9188218136908217) > weight (0.25) 2024-09-15T01:08:50.893 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-15T01:08:57.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:08:57.007+0000 7f6a1a2be640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d676403c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=190 2024-09-15T01:08:57.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:08:57.007+0000 7f6a1a2be640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d676403c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=190, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=1108ad63-b7b4-4889-ae31-b1fd6de8431b, primary_snap_id=be, last_copied_object_number=128, snap_seqs={190=18446744073709551614}] 2024-09-15T01:08:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring disabled 2024-09-15T01:08:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-15T01:08:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:08:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:08:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:08:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:08:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:08:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_6 2024-09-15T01:08:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:08:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:08:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_6 2024-09-15T01:08:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-15T01:08:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:08:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-15T01:08:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:08:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:08:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:08:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-15T01:08:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:08:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:08:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:08:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:08:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:08:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_6 2024-09-15T01:08:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:08:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:08:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_6 2024-09-15T01:08:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-15T01:08:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:08:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-15T01:08:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:08:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:08:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:08:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:08:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-15T01:08:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-15T01:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-15T01:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_6 2024-09-15T01:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_6 2024-09-15T01:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-15T01:08:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-15T01:08:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T01:08:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:08:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T01:08:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_7 2024-09-15T01:08:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-15T01:08:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:08:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:08:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:08:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:08:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T01:08:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T01:08:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T01:08:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:08:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-15T01:08:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-15T01:08:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:08:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T01:08:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-15T01:08:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T01:08:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-15T01:08:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_7': 2024-09-15T01:08:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T01:08:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:08:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T01:08:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 19f155f0e77b 2024-09-15T01:08:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.19f155f0e77b 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 01:08:57 2024 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 01:08:57 2024 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 01:08:57 2024 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: e3201c5e-6a07-4451-bb9f-f990fc2e13b6 2024-09-15T01:08:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T01:08:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-15T01:08:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:08:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:08:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:08:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T01:08:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=7 2024-09-15T01:08:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:08:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-15T01:08:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 37844992 bytes 2024-09-15T01:08:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 73850880 bytes 2024-09-15T01:08:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 125270016 bytes 2024-09-15T01:08:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 188214272 bytes 2024-09-15T01:08:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 207039488 bytes 2024-09-15T01:08:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 277338112 bytes 2024-09-15T01:08:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 354683904 bytes 2024-09-15T01:08:59.197 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:08:59.195+0000 7fc075401640 -1 reset not still connected to 0x55a0ba953860 2024-09-15T01:08:59.197 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:08:59.195+0000 7fc075401640 -1 reset not still connected to 0x55a0bd91dee0 2024-09-15T01:08:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 370392064 bytes 2024-09-15T01:08:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 417419264 bytes 2024-09-15T01:08:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 465849344 bytes 2024-09-15T01:08:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 473693184 bytes 2024-09-15T01:08:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 506890240 bytes 2024-09-15T01:08:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 574902272 bytes 2024-09-15T01:08:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 608771072 bytes 2024-09-15T01:09:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 676143104 bytes 2024-09-15T01:09:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 722935808 bytes 2024-09-15T01:09:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 764205056 bytes 2024-09-15T01:09:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 807365632 bytes 2024-09-15T01:09:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 902220800 bytes 2024-09-15T01:09:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 932307968 bytes 2024-09-15T01:09:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 973888512 bytes 2024-09-15T01:09:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1004965888 bytes 2024-09-15T01:09:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1013312512 bytes 2024-09-15T01:09:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1013923840 bytes 2024-09-15T01:09:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1104613376 bytes 2024-09-15T01:09:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1195045888 bytes 2024-09-15T01:09:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1228753920 bytes 2024-09-15T01:09:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1234546688 bytes 2024-09-15T01:09:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1298930688 bytes 2024-09-15T01:09:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1348765696 bytes 2024-09-15T01:09:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1365471232 bytes 2024-09-15T01:09:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1446054912 bytes 2024-09-15T01:09:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1459362816 bytes 2024-09-15T01:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1459970048 bytes 2024-09-15T01:09:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1466099712 bytes 2024-09-15T01:09:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1574168576 bytes 2024-09-15T01:09:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1612094464 bytes 2024-09-15T01:09:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1691378688 bytes 2024-09-15T01:09:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1739490304 bytes 2024-09-15T01:09:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1760543744 bytes 2024-09-15T01:09:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1786030080 bytes 2024-09-15T01:09:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1822934016 bytes 2024-09-15T01:09:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1841613824 bytes 2024-09-15T01:09:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1850786816 bytes 2024-09-15T01:09:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1881276416 bytes 2024-09-15T01:09:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1916200960 bytes 2024-09-15T01:09:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1948088320 bytes 2024-09-15T01:09:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2019547136 bytes 2024-09-15T01:09:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2069048320 bytes 2024-09-15T01:09:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2115303424 bytes 2024-09-15T01:09:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2166774784 bytes 2024-09-15T01:09:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2222322688 bytes 2024-09-15T01:09:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2323535872 bytes 2024-09-15T01:09:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2343795712 bytes 2024-09-15T01:09:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T01:09:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:09:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T01:09:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:09:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-15T01:09:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:09:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:09:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=present 2024-09-15T01:09:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:09:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:09:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:09:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_7 2024-09-15T01:09:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:09:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:09:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_7 2024-09-15T01:09:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-15T01:09:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:09:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id=1625747060f7 2024-09-15T01:09:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:09:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:09:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_7 present 1625747060f7 2024-09-15T01:09:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:09:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:09:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=present 2024-09-15T01:09:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id=1625747060f7 2024-09-15T01:09:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:09:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:09:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_7 2024-09-15T01:09:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:09:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:09:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_7 2024-09-15T01:09:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-15T01:09:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:09:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id=1625747060f7 2024-09-15T01:09:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1625747060f7 2024-09-15T01:09:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -z 1625747060f7 -o 1625747060f7 = 1625747060f7 2024-09-15T01:09:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=present 2024-09-15T01:09:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test present = present 2024-09-15T01:09:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:09:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-15T01:09:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:09:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:09:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-15T01:09:12.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:09:12.241+0000 7f0d7c902640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ca804e2d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:09:12.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:09:12.241+0000 7f0d83910640 -1 rbd::mirror::ImageReplayer: 0x55ca80ef4500 [3/e3201c5e-6a07-4451-bb9f-f990fc2e13b6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:09:33.100 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-15T01:09:33.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-15T01:09:33.101 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-15T01:09:33.101 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-15T01:09:33.102 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:09:33.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:09:33.100+0000 7f79ab470640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 69194) UID: 1000 2024-09-15T01:09:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring disabled 2024-09-15T01:09:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-15T01:09:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:09:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:09:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:09:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:09:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:09:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:09:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_7 2024-09-15T01:09:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:09:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:09:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_7 2024-09-15T01:09:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-15T01:09:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:09:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-15T01:09:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:09:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:09:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:09:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-15T01:09:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:09:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:09:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:09:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:09:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:09:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:09:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_7 2024-09-15T01:09:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:09:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:09:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_7 2024-09-15T01:09:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-15T01:09:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:09:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-15T01:09:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:09:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:09:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:09:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:09:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-15T01:09:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:09:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:09:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-15T01:09:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-15T01:09:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:09:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:09:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:09:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:09:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_7 2024-09-15T01:09:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:09:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_7 2024-09-15T01:09:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-15T01:09:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-15T01:09:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_8 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T01:09:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T01:09:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T01:09:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:09:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-15T01:09:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-15T01:09:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:09:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:09:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T01:09:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-15T01:09:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T01:09:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-15T01:09:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_8': 2024-09-15T01:09:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T01:09:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:09:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T01:09:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1a0511aa8402 2024-09-15T01:09:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1a0511aa8402 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 01:09:35 2024 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 01:09:35 2024 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 01:09:35 2024 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 3493508b-91e4-4d59-9be4-bd2aa21c9b06 2024-09-15T01:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T01:09:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-15T01:09:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:09:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:09:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T01:09:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=11 2024-09-15T01:09:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:09:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-15T01:09:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 19749888 bytes 2024-09-15T01:09:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 33140736 bytes 2024-09-15T01:09:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 94508032 bytes 2024-09-15T01:09:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 166333440 bytes 2024-09-15T01:09:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 205233152 bytes 2024-09-15T01:09:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 271655936 bytes 2024-09-15T01:09:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 316013568 bytes 2024-09-15T01:09:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 334502912 bytes 2024-09-15T01:09:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 381596672 bytes 2024-09-15T01:09:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 445465600 bytes 2024-09-15T01:09:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 543666176 bytes 2024-09-15T01:09:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 599081984 bytes 2024-09-15T01:09:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 629303296 bytes 2024-09-15T01:09:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 666910720 bytes 2024-09-15T01:09:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 679962624 bytes 2024-09-15T01:09:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 700887040 bytes 2024-09-15T01:09:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 735126528 bytes 2024-09-15T01:09:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 760335360 bytes 2024-09-15T01:09:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 828095488 bytes 2024-09-15T01:09:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 878295040 bytes 2024-09-15T01:09:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1021976576 bytes 2024-09-15T01:09:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1043808256 bytes 2024-09-15T01:09:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1059231744 bytes 2024-09-15T01:09:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1114406912 bytes 2024-09-15T01:09:39.204 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-15T01:09:39.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-15T01:09:39.205 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-15T01:09:39.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-15T01:09:39.208 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-09-15T01:09:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1115019264 bytes 2024-09-15T01:09:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1164691456 bytes 2024-09-15T01:09:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1242909696 bytes 2024-09-15T01:09:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1293047808 bytes 2024-09-15T01:09:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1314997248 bytes 2024-09-15T01:09:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1360490496 bytes 2024-09-15T01:09:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1391477760 bytes 2024-09-15T01:09:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1421804544 bytes 2024-09-15T01:09:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1442833408 bytes 2024-09-15T01:09:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1465033728 bytes 2024-09-15T01:09:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1524947968 bytes 2024-09-15T01:09:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1591090176 bytes 2024-09-15T01:09:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1603452928 bytes 2024-09-15T01:09:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1605025792 bytes 2024-09-15T01:09:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1684167680 bytes 2024-09-15T01:09:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1734541312 bytes 2024-09-15T01:09:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1769721856 bytes 2024-09-15T01:09:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1800858624 bytes 2024-09-15T01:09:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1801713664 bytes 2024-09-15T01:09:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1802334208 bytes 2024-09-15T01:09:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1826568192 bytes 2024-09-15T01:09:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1827189760 bytes 2024-09-15T01:09:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1864873984 bytes 2024-09-15T01:09:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1873004544 bytes 2024-09-15T01:09:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1919856640 bytes 2024-09-15T01:09:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1962853376 bytes 2024-09-15T01:09:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2057422848 bytes 2024-09-15T01:09:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2120065024 bytes 2024-09-15T01:09:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2173189120 bytes 2024-09-15T01:09:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2210114560 bytes 2024-09-15T01:09:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2257574912 bytes 2024-09-15T01:09:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2324632576 bytes 2024-09-15T01:09:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2414045184 bytes 2024-09-15T01:09:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2427722752 bytes 2024-09-15T01:09:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2524953600 bytes 2024-09-15T01:09:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2597976064 bytes 2024-09-15T01:09:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2606036992 bytes 2024-09-15T01:09:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2674761728 bytes 2024-09-15T01:09:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2718401536 bytes 2024-09-15T01:09:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2771728384 bytes 2024-09-15T01:09:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2817452032 bytes 2024-09-15T01:09:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2856577024 bytes 2024-09-15T01:09:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2866749440 bytes 2024-09-15T01:09:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2906404864 bytes 2024-09-15T01:09:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2931651584 bytes 2024-09-15T01:09:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2938487808 bytes 2024-09-15T01:09:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2951175168 bytes 2024-09-15T01:09:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3011672064 bytes 2024-09-15T01:09:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3069984768 bytes 2024-09-15T01:09:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3110402048 bytes 2024-09-15T01:09:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3181051904 bytes 2024-09-15T01:09:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3247712256 bytes 2024-09-15T01:09:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3332212736 bytes 2024-09-15T01:09:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3357576192 bytes 2024-09-15T01:09:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3443391488 bytes 2024-09-15T01:09:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3509043200 bytes 2024-09-15T01:09:46.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3600704512 bytes 2024-09-15T01:09:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3631582208 bytes 2024-09-15T01:09:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3647156224 bytes 2024-09-15T01:09:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3654286336 bytes 2024-09-15T01:09:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3672668160 bytes 2024-09-15T01:09:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3715610624 bytes 2024-09-15T01:09:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3800918016 bytes 2024-09-15T01:09:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3886251008 bytes 2024-09-15T01:09:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T01:09:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:09:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T01:09:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:09:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-15T01:09:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:09:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:09:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=present 2024-09-15T01:09:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:09:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:09:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:09:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_8 2024-09-15T01:09:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:09:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:09:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_8 2024-09-15T01:09:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-15T01:09:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:09:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id=16259cefb51c 2024-09-15T01:09:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:09:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:09:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_8 present 16259cefb51c 2024-09-15T01:09:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:09:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:09:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=present 2024-09-15T01:09:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id=16259cefb51c 2024-09-15T01:09:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:09:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:09:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_8 2024-09-15T01:09:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:09:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:09:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_8 2024-09-15T01:09:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-15T01:09:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:09:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id=16259cefb51c 2024-09-15T01:09:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 16259cefb51c 2024-09-15T01:09:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -z 16259cefb51c -o 16259cefb51c = 16259cefb51c 2024-09-15T01:09:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=present 2024-09-15T01:09:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test present = present 2024-09-15T01:09:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:09:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-15T01:09:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:09:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:09:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:09:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-15T01:09:58.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7961089242068834) > weight (0.25) 2024-09-15T01:09:58.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4102095033028368) > weight (0.25) 2024-09-15T01:09:58.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.34239436454108574) > weight (0.25) 2024-09-15T01:09:58.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.34642587892037646) > weight (0.25) 2024-09-15T01:09:58.895 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-09-15T01:10:12.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:10:12.242+0000 7f0d7d103640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ca8019fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:10:12.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:10:12.242+0000 7f0d8310f640 -1 rbd::mirror::ImageReplayer: 0x55ca80ef4500 [3/3493508b-91e4-4d59-9be4-bd2aa21c9b06] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:10:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring disabled 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_8 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:10:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_8 2024-09-15T01:10:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-15T01:10:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:10:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-15T01:10:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:10:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:10:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:10:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-15T01:10:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:10:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:10:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:10:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:10:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:10:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:10:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_8 2024-09-15T01:10:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:10:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:10:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_8 2024-09-15T01:10:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-15T01:10:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:10:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-15T01:10:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:10:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:10:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:10:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:10:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-15T01:10:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:10:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:10:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-15T01:10:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-15T01:10:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:10:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:10:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:10:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:10:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_8 2024-09-15T01:10:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:10:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_8 2024-09-15T01:10:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-15T01:10:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-15T01:10:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T01:10:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:10:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T01:10:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_9 2024-09-15T01:10:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:10:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-15T01:10:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-15T01:10:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:10:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:10:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T01:10:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-15T01:10:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T01:10:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-15T01:10:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_9': 2024-09-15T01:10:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T01:10:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:10:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T01:10:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1a1f9b3534f 2024-09-15T01:10:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1a1f9b3534f 2024-09-15T01:10:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:10:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T01:10:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:10:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:10:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 01:10:18 2024 2024-09-15T01:10:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 01:10:18 2024 2024-09-15T01:10:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 01:10:18 2024 2024-09-15T01:10:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:10:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:10:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 90573912-77ec-4a01-b328-86ef400bd129 2024-09-15T01:10:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T01:10:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-15T01:10:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:10:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:10:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T01:10:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=12 2024-09-15T01:10:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:10:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-15T01:10:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 65703936 bytes 2024-09-15T01:10:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 152589312 bytes 2024-09-15T01:10:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 167770112 bytes 2024-09-15T01:10:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 201354240 bytes 2024-09-15T01:10:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 284769280 bytes 2024-09-15T01:10:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 297524224 bytes 2024-09-15T01:10:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 313651200 bytes 2024-09-15T01:10:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 379338752 bytes 2024-09-15T01:10:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 461635584 bytes 2024-09-15T01:10:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 501984256 bytes 2024-09-15T01:10:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 603587584 bytes 2024-09-15T01:10:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 726575104 bytes 2024-09-15T01:10:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 757278720 bytes 2024-09-15T01:10:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 815525888 bytes 2024-09-15T01:10:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 870423552 bytes 2024-09-15T01:10:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 895273984 bytes 2024-09-15T01:10:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 937178112 bytes 2024-09-15T01:10:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 997308416 bytes 2024-09-15T01:10:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 1017997312 bytes 2024-09-15T01:10:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 1035125760 bytes 2024-09-15T01:10:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 1045423104 bytes 2024-09-15T01:10:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 1089474560 bytes 2024-09-15T01:10:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 1115917312 bytes 2024-09-15T01:10:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 1224129536 bytes 2024-09-15T01:10:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 1271800832 bytes 2024-09-15T01:10:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1294028800 bytes 2024-09-15T01:10:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1335641088 bytes 2024-09-15T01:10:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1378399232 bytes 2024-09-15T01:10:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1445016576 bytes 2024-09-15T01:10:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1488587776 bytes 2024-09-15T01:10:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1586100224 bytes 2024-09-15T01:10:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1614785536 bytes 2024-09-15T01:10:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1662763008 bytes 2024-09-15T01:10:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1668822016 bytes 2024-09-15T01:10:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1689278464 bytes 2024-09-15T01:10:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1694260224 bytes 2024-09-15T01:10:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1719621632 bytes 2024-09-15T01:10:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1784549376 bytes 2024-09-15T01:10:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1857315840 bytes 2024-09-15T01:10:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1890075648 bytes 2024-09-15T01:10:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1942899712 bytes 2024-09-15T01:10:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 2000129024 bytes 2024-09-15T01:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 2012636160 bytes 2024-09-15T01:10:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 2085441536 bytes 2024-09-15T01:10:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 2138726400 bytes 2024-09-15T01:10:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 2155757568 bytes 2024-09-15T01:10:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 2180332544 bytes 2024-09-15T01:10:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 2228540416 bytes 2024-09-15T01:10:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 2263881728 bytes 2024-09-15T01:10:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 2321510400 bytes 2024-09-15T01:10:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2347411456 bytes 2024-09-15T01:10:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2399297536 bytes 2024-09-15T01:10:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2427752448 bytes 2024-09-15T01:10:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2446432256 bytes 2024-09-15T01:10:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2459334656 bytes 2024-09-15T01:10:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2459957248 bytes 2024-09-15T01:10:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2555702272 bytes 2024-09-15T01:10:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2578436096 bytes 2024-09-15T01:10:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2613366784 bytes 2024-09-15T01:10:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2645259264 bytes 2024-09-15T01:10:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2687804416 bytes 2024-09-15T01:10:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2799024128 bytes 2024-09-15T01:10:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2857955328 bytes 2024-09-15T01:10:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2902894592 bytes 2024-09-15T01:10:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2911075328 bytes 2024-09-15T01:10:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2932483072 bytes 2024-09-15T01:10:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 3003422720 bytes 2024-09-15T01:10:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 3013011456 bytes 2024-09-15T01:10:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 3032009728 bytes 2024-09-15T01:10:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 3050109952 bytes 2024-09-15T01:10:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 3050741760 bytes 2024-09-15T01:10:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 3130352640 bytes 2024-09-15T01:10:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 3144626176 bytes 2024-09-15T01:10:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 3200699392 bytes 2024-09-15T01:10:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 3280342016 bytes 2024-09-15T01:10:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 3280939008 bytes 2024-09-15T01:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 3300694016 bytes 2024-09-15T01:10:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 3352729600 bytes 2024-09-15T01:10:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 3386304512 bytes 2024-09-15T01:10:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3409577984 bytes 2024-09-15T01:10:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3453721600 bytes 2024-09-15T01:10:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8200 IOs, 3479727104 bytes 2024-09-15T01:10:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8300 IOs, 3545335808 bytes 2024-09-15T01:10:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8400 IOs, 3604644864 bytes 2024-09-15T01:10:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8500 IOs, 3668920320 bytes 2024-09-15T01:10:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8600 IOs, 3680748544 bytes 2024-09-15T01:10:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8700 IOs, 3717726208 bytes 2024-09-15T01:10:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8800 IOs, 3728615424 bytes 2024-09-15T01:10:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8900 IOs, 3751657472 bytes 2024-09-15T01:10:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9000 IOs, 3782453248 bytes 2024-09-15T01:10:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9100 IOs, 3784759296 bytes 2024-09-15T01:10:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9200 IOs, 3790004224 bytes 2024-09-15T01:10:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9300 IOs, 3790797824 bytes 2024-09-15T01:10:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9400 IOs, 3884907520 bytes 2024-09-15T01:10:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9500 IOs, 3885492224 bytes 2024-09-15T01:10:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9600 IOs, 3979193344 bytes 2024-09-15T01:10:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:9700 IOs, 3996275712 bytes 2024-09-15T01:10:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T01:10:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:10:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T01:10:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:10:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-15T01:10:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:10:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:10:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=present 2024-09-15T01:10:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:10:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:10:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:10:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_9 2024-09-15T01:10:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:10:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:10:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_9 2024-09-15T01:10:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-15T01:10:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id=16257f3762b6 2024-09-15T01:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:10:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_9 present 16257f3762b6 2024-09-15T01:10:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:10:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:10:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=present 2024-09-15T01:10:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id=16257f3762b6 2024-09-15T01:10:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:10:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:10:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_9 2024-09-15T01:10:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:10:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:10:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_9 2024-09-15T01:10:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-15T01:10:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:10:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id=16257f3762b6 2024-09-15T01:10:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 16257f3762b6 2024-09-15T01:10:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -z 16257f3762b6 -o 16257f3762b6 = 16257f3762b6 2024-09-15T01:10:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=present 2024-09-15T01:10:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test present = present 2024-09-15T01:10:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:10:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-15T01:10:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:10:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:10:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:10:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-15T01:10:42.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:10:42.242+0000 7f0d7c902640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ca8019fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:10:42.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:10:42.242+0000 7f0d8310f640 -1 rbd::mirror::ImageReplayer: 0x55ca80ef4500 [3/90573912-77ec-4a01-b328-86ef400bd129] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:11:00.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:00.955+0000 7f6a19abd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d676388000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=193 2024-09-15T01:11:00.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:00.955+0000 7f6a19abd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d676388000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=193, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=1108ad63-b7b4-4889-ae31-b1fd6de8431b, primary_snap_id=c1, last_copied_object_number=128, snap_seqs={193=18446744073709551614}] 2024-09-15T01:11:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring disabled 2024-09-15T01:11:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-15T01:11:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:11:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:11:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:11:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:11:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:11:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_9 2024-09-15T01:11:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_9 2024-09-15T01:11:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-15T01:11:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:11:00.987 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:11:00.983+0000 7fc075401640 -1 reset not still connected to 0x55a0ba953860 2024-09-15T01:11:00.987 INFO:tasks.ceph.cluster1.osd.2.smithi002.stderr:2024-09-15T01:11:00.983+0000 7fc075401640 -1 reset not still connected to 0x55a0bd91dee0 2024-09-15T01:11:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-15T01:11:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:11:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:11:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_9 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_9 2024-09-15T01:11:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-15T01:11:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:11:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-15T01:11:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:11:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:11:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:11:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-15T01:11:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:11:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-15T01:11:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-15T01:11:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:11:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:11:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:11:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_9 2024-09-15T01:11:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_9 2024-09-15T01:11:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-15T01:11:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-15T01:11:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T01:11:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for i in `seq 1 10` 2024-09-15T01:11:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image=image_10 2024-09-15T01:11:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-15T01:11:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:11:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:11:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:11:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local size=128 2024-09-15T01:11:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -n 512M ']' 2024-09-15T01:11:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ size=512M 2024-09-15T01:11:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ shift 2024-09-15T01:11:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-15T01:11:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-15T01:11:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local mode=snapshot 2024-09-15T01:11:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-15T01:11:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring enabled 2024-09-15T01:11:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-15T01:11:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:rbd image 'image_10': 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: size 512 MiB in 128 objects 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: order 22 (4 MiB objects) 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: snapshot_count: 1 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: id: 1a2ff42e55b4 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: block_name_prefix: rbd_data.1a2ff42e55b4 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: format: 2 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: op_features: 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: flags: 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: create_timestamp: Sun Sep 15 01:11:01 2024 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: access_timestamp: Sun Sep 15 01:11:01 2024 2024-09-15T01:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: modify_timestamp: Sun Sep 15 01:11:01 2024 2024-09-15T01:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring state: enabled 2024-09-15T01:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring mode: snapshot 2024-09-15T01:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring global id: 2c422a6a-bd6a-41ca-8e4a-6011b3bea052 2024-09-15T01:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout: mirroring primary: true 2024-09-15T01:11:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-15T01:11:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-15T01:11:02.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local duration=10 2024-09-15T01:11:02.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:11:02.020 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-15T01:11:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:100 IOs, 100222976 bytes 2024-09-15T01:11:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:200 IOs, 125537280 bytes 2024-09-15T01:11:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:300 IOs, 152840192 bytes 2024-09-15T01:11:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:400 IOs, 202070016 bytes 2024-09-15T01:11:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:500 IOs, 249551872 bytes 2024-09-15T01:11:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:600 IOs, 278512640 bytes 2024-09-15T01:11:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:700 IOs, 328914944 bytes 2024-09-15T01:11:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:800 IOs, 332200960 bytes 2024-09-15T01:11:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:900 IOs, 396783616 bytes 2024-09-15T01:11:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1000 IOs, 414861312 bytes 2024-09-15T01:11:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1100 IOs, 478962688 bytes 2024-09-15T01:11:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1200 IOs, 528856064 bytes 2024-09-15T01:11:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1300 IOs, 553950208 bytes 2024-09-15T01:11:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1400 IOs, 560012288 bytes 2024-09-15T01:11:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1500 IOs, 577359872 bytes 2024-09-15T01:11:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1600 IOs, 630167552 bytes 2024-09-15T01:11:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1700 IOs, 667909120 bytes 2024-09-15T01:11:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1800 IOs, 721180672 bytes 2024-09-15T01:11:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:1900 IOs, 785982464 bytes 2024-09-15T01:11:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2000 IOs, 837426176 bytes 2024-09-15T01:11:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2100 IOs, 862020608 bytes 2024-09-15T01:11:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2200 IOs, 910393344 bytes 2024-09-15T01:11:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2300 IOs, 938793984 bytes 2024-09-15T01:11:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2400 IOs, 974977024 bytes 2024-09-15T01:11:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2500 IOs, 996994048 bytes 2024-09-15T01:11:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2600 IOs, 1018803200 bytes 2024-09-15T01:11:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2700 IOs, 1047612416 bytes 2024-09-15T01:11:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2800 IOs, 1085722624 bytes 2024-09-15T01:11:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:2900 IOs, 1238880256 bytes 2024-09-15T01:11:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3000 IOs, 1260787712 bytes 2024-09-15T01:11:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3100 IOs, 1307536384 bytes 2024-09-15T01:11:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3200 IOs, 1412291584 bytes 2024-09-15T01:11:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3300 IOs, 1472775168 bytes 2024-09-15T01:11:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3400 IOs, 1506017280 bytes 2024-09-15T01:11:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3500 IOs, 1562637312 bytes 2024-09-15T01:11:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3600 IOs, 1563265024 bytes 2024-09-15T01:11:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3700 IOs, 1585757184 bytes 2024-09-15T01:11:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3800 IOs, 1608965120 bytes 2024-09-15T01:11:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:3900 IOs, 1659651072 bytes 2024-09-15T01:11:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4000 IOs, 1673676800 bytes 2024-09-15T01:11:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4100 IOs, 1706903552 bytes 2024-09-15T01:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4200 IOs, 1752537088 bytes 2024-09-15T01:11:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4300 IOs, 1800271872 bytes 2024-09-15T01:11:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4400 IOs, 1835258880 bytes 2024-09-15T01:11:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4500 IOs, 1880919040 bytes 2024-09-15T01:11:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4600 IOs, 1922179072 bytes 2024-09-15T01:11:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4700 IOs, 1935268864 bytes 2024-09-15T01:11:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4800 IOs, 1960999936 bytes 2024-09-15T01:11:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:4900 IOs, 1974480896 bytes 2024-09-15T01:11:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5000 IOs, 1975074816 bytes 2024-09-15T01:11:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5100 IOs, 2046033920 bytes 2024-09-15T01:11:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5200 IOs, 2049918976 bytes 2024-09-15T01:11:08.605 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5300 IOs, 2051267584 bytes 2024-09-15T01:11:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5400 IOs, 2135878656 bytes 2024-09-15T01:11:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5500 IOs, 2156249088 bytes 2024-09-15T01:11:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5600 IOs, 2182494208 bytes 2024-09-15T01:11:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5700 IOs, 2267067392 bytes 2024-09-15T01:11:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5800 IOs, 2286479360 bytes 2024-09-15T01:11:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:5900 IOs, 2303254528 bytes 2024-09-15T01:11:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6000 IOs, 2357135360 bytes 2024-09-15T01:11:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6100 IOs, 2422139904 bytes 2024-09-15T01:11:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6200 IOs, 2478103552 bytes 2024-09-15T01:11:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6300 IOs, 2488525824 bytes 2024-09-15T01:11:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6400 IOs, 2504647680 bytes 2024-09-15T01:11:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6500 IOs, 2547894272 bytes 2024-09-15T01:11:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6600 IOs, 2570340352 bytes 2024-09-15T01:11:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6700 IOs, 2592918528 bytes 2024-09-15T01:11:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6800 IOs, 2616114176 bytes 2024-09-15T01:11:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:6900 IOs, 2705288192 bytes 2024-09-15T01:11:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7000 IOs, 2741664768 bytes 2024-09-15T01:11:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7100 IOs, 2752100352 bytes 2024-09-15T01:11:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7200 IOs, 2752676864 bytes 2024-09-15T01:11:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7300 IOs, 2767214592 bytes 2024-09-15T01:11:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7400 IOs, 2840828928 bytes 2024-09-15T01:11:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7500 IOs, 2901880832 bytes 2024-09-15T01:11:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7600 IOs, 2953876480 bytes 2024-09-15T01:11:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7700 IOs, 2954485760 bytes 2024-09-15T01:11:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7800 IOs, 2977260544 bytes 2024-09-15T01:11:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:7900 IOs, 2998612992 bytes 2024-09-15T01:11:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8000 IOs, 3067952128 bytes 2024-09-15T01:11:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:8100 IOs, 3087903744 bytes 2024-09-15T01:11:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=124 2024-09-15T01:11:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:11:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 124 -eq 124 ']' 2024-09-15T01:11:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=present 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_10 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_10 2024-09-15T01:11:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-15T01:11:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:11:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id=16251a008b89 2024-09-15T01:11:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:11:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:11:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_10 present 16251a008b89 2024-09-15T01:11:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=present 2024-09-15T01:11:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id=16251a008b89 2024-09-15T01:11:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:11:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:11:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_10 2024-09-15T01:11:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_10 2024-09-15T01:11:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-15T01:11:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:11:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id=16251a008b89 2024-09-15T01:11:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 16251a008b89 2024-09-15T01:11:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -z 16251a008b89 -o 16251a008b89 = 16251a008b89 2024-09-15T01:11:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_state=present 2024-09-15T01:11:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test present = present 2024-09-15T01:11:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-15T01:11:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-15T01:11:18.895 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-15T01:11:18.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-15T01:11:18.896 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-15T01:11:18.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:18.895+0000 7f6a27ad9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 64804) UID: 1000 2024-09-15T01:11:20.897 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-15T01:11:20.897 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:11:20.898 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-15T01:11:20.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-15T01:11:20.898 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-15T01:11:20.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-15T01:11:20.901 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-09-15T01:11:22.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:22.987+0000 7f0d7c902640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ca804e2ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:11:22.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:22.987+0000 7f0d83910640 -1 rbd::mirror::ImageReplayer: 0x55ca80ef4780 [3/2c422a6a-bd6a-41ca-8e4a-6011b3bea052] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:11:24.209 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5321065716349689) > weight (0.25) 2024-09-15T01:11:24.209 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7580486202133658) > weight (0.25) 2024-09-15T01:11:24.209 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3714761393223762) > weight (0.25) 2024-09-15T01:11:24.209 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-15T01:11:24.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-15T01:11:24.210 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-15T01:11:24.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:24.209+0000 7f83e4949640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 61205) UID: 1000 2024-09-15T01:11:27.210 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-15T01:11:27.211 INFO:teuthology.orchestra.run:waiting for 600 2024-09-15T01:11:27.211 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-15T01:11:27.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-15T01:11:27.212 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-15T01:11:27.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-15T01:11:27.215 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-15T01:11:42.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:42.242+0000 7f0d7c902640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ca80799860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-15T01:11:42.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:42.242+0000 7f0d8310f640 -1 rbd::mirror::ImageReplayer: 0x55ca80ef4780 [3/2c422a6a-bd6a-41ca-8e4a-6011b3bea052] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-15T01:11:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:Mirroring disabled 2024-09-15T01:11:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-15T01:11:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local state=deleted 2024-09-15T01:11:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:11:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local s 2024-09-15T01:11:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:11:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_10 2024-09-15T01:11:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_10 2024-09-15T01:11:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-15T01:11:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:11:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-15T01:11:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ image_id= 2024-09-15T01:11:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-15T01:11:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0.1 2024-09-15T01:11:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-15T01:11:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local test_state=deleted 2024-09-15T01:11:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image_id= 2024-09-15T01:11:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_state=deleted 2024-09-15T01:11:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local current_image_id 2024-09-15T01:11:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ get_image_id cluster1 mirror image_10 2024-09-15T01:11:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local image=image_10 2024-09-15T01:11:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-15T01:11:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-15T01:11:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-15T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ current_image_id= 2024-09-15T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n '' 2024-09-15T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test deleted = deleted 2024-09-15T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-15T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-15T01:11:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-15T01:11:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-15T01:11:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:11:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ remove_image cluster2 mirror image_10 2024-09-15T01:11:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local image=image_10 2024-09-15T01:11:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-15T01:11:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-15T01:11:43.463 INFO:tasks.ceph.cluster2.osd.0.smithi059.stderr:2024-09-15T01:11:43.461+0000 7f173c535640 -1 reset not still connected to 0x5647147ca410 2024-09-15T01:11:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-15T01:11:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-15T01:11:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ tee -a /tmp/tmp.nS85ynkalX/rbd-mirror.test.log 2024-09-15T01:11:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ date '+%F %T' 2024-09-15T01:11:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo 2024-09-15 01:11:43 TEST: check if removed 'images'\''' OMAP are removed 2024-09-15T01:11:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:2024-09-15 01:11:43 TEST: check if removed images' OMAP are removed 2024-09-15T01:11:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-15T01:11:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-15T01:11:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local obj_name=rbd_mirroring 2024-09-15T01:11:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local filter=status_global 2024-09-15T01:11:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-15T01:11:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:11:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:11:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-15T01:11:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirroring 2024-09-15T01:11:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local filter=status_global 2024-09-15T01:11:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-15T01:11:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirroring 2024-09-15T01:11:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-15T01:11:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -c status_global 2024-09-15T01:11:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 = 0 2024-09-15T01:11:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=0 2024-09-15T01:11:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:11:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 0 ']' 2024-09-15T01:11:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-15T01:11:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local obj_name=rbd_mirroring 2024-09-15T01:11:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local filter=image_ 2024-09-15T01:11:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-15T01:11:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:11:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:11:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-15T01:11:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirroring 2024-09-15T01:11:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local filter=image_ 2024-09-15T01:11:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-15T01:11:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirroring 2024-09-15T01:11:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-15T01:11:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -c image_ 2024-09-15T01:11:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 = 0 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=0 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 0 ']' 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local obj_name=rbd_mirror_leader 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local filter=image_map 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:11:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:11:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-15T01:11:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirror_leader 2024-09-15T01:11:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local filter=image_map 2024-09-15T01:11:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-15T01:11:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirror_leader 2024-09-15T01:11:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -c image_map 2024-09-15T01:11:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-15T01:11:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 1 = 0 2024-09-15T01:11:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=1 2024-09-15T01:11:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:11:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 1 -eq 0 ']' 2024-09-15T01:11:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-15T01:11:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 1 2024-09-15T01:11:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:11:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-15T01:11:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirror_leader 2024-09-15T01:11:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local filter=image_map 2024-09-15T01:11:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-15T01:11:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirror_leader 2024-09-15T01:11:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -c image_map 2024-09-15T01:11:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-15T01:11:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 1 = 0 2024-09-15T01:11:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=1 2024-09-15T01:11:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:11:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 1 -eq 0 ']' 2024-09-15T01:11:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-15T01:11:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 2 2024-09-15T01:11:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:11:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-15T01:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirror_leader 2024-09-15T01:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local filter=image_map 2024-09-15T01:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-15T01:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster1 2024-09-15T01:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirror_leader 2024-09-15T01:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-15T01:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -c image_map 2024-09-15T01:11:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 = 0 2024-09-15T01:11:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=0 2024-09-15T01:11:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:11:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 0 ']' 2024-09-15T01:11:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-15T01:11:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-15T01:11:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local obj_name=rbd_mirroring 2024-09-15T01:11:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local filter=status_global 2024-09-15T01:11:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-15T01:11:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:11:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:11:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-15T01:11:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster2 2024-09-15T01:11:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirroring 2024-09-15T01:11:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local filter=status_global 2024-09-15T01:11:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-15T01:11:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster2 2024-09-15T01:11:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirroring 2024-09-15T01:11:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -c status_global 2024-09-15T01:11:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-15T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 = 0 2024-09-15T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=0 2024-09-15T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 0 ']' 2024-09-15T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-15T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local obj_name=rbd_mirroring 2024-09-15T01:11:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local filter=image_ 2024-09-15T01:11:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-15T01:11:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:11:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster2 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirroring 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local filter=image_ 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster2 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -c image_ 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirroring 2024-09-15T01:11:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-15T01:11:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 = 0 2024-09-15T01:11:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=0 2024-09-15T01:11:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:11:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 0 ']' 2024-09-15T01:11:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-15T01:11:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local pool=mirror 2024-09-15T01:11:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local obj_name=rbd_mirror_leader 2024-09-15T01:11:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local filter=image_map 2024-09-15T01:11:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-15T01:11:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ sleep 0 2024-09-15T01:11:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:11:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-15T01:11:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster2 2024-09-15T01:11:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirror_leader 2024-09-15T01:11:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local filter=image_map 2024-09-15T01:11:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-15T01:11:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local cluster=cluster2 2024-09-15T01:11:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local pool=mirror 2024-09-15T01:11:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ local obj_name=rbd_mirror_leader 2024-09-15T01:11:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ grep -c image_map 2024-09-15T01:11:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-15T01:11:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test 0 = 0 2024-09-15T01:11:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ error_code=0 2024-09-15T01:11:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set -e 2024-09-15T01:11:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 0 ']' 2024-09-15T01:11:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 0 2024-09-15T01:11:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ cleanup 0 2024-09-15T01:11:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local error_code=0 2024-09-15T01:11:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ set +e 2024-09-15T01:11:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -ne 0 ']' 2024-09-15T01:11:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -z '' ']' 2024-09-15T01:11:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster instance 2024-09-15T01:11:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T01:11:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-15T01:11:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:pool 'mirror' does not exist 2024-09-15T01:11:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T01:11:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-15T01:11:48.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:48.944+0000 7f85a684f640 -1 rbd::mirror::InstanceWatcher: 0x55f9d61aca88 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:48.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:48.971+0000 7f85ad05c640 -1 rbd::mirror::LeaderWatcher: 0x55f9d5331800 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-15T01:11:48.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:48.971+0000 7f85a684f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f9d5a51b00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-15T01:11:48.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:48.971+0000 7f85a684f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f9d5a50900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-15T01:11:48.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:48.971+0000 7f85a684f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f9d5544480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-15T01:11:48.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:48.971+0000 7f85a604e640 -1 rbd::mirror::PoolWatcher: 0x55f9d5b5f2c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:48.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:48.971+0000 7f85a604e640 -1 rbd::mirror::PoolWatcher: 0x55f9d5b5fc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:48.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:48.971+0000 7f85a684f640 -1 rbd::mirror::PoolWatcher: 0x55f9d60612c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:48.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:48.975+0000 7f85ac85b640 -1 rbd::mirror::LeaderWatcher: 0x55f9d5331800 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-15T01:11:48.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:48.975+0000 7f85ac85b640 -1 rbd::mirror::LeaderWatcher: 0x55f9d5331800 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-15T01:11:49.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.096+0000 7f94432f0640 -1 rbd::mirror::InstanceWatcher: 0x562c1b562380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5991 object: (2) No such file or directory 2024-09-15T01:11:49.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.096+0000 7f9443af1640 -1 rbd::mirror::InstanceWatcher: 0x562c1b562a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5991 object: (2) No such file or directory 2024-09-15T01:11:49.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.097+0000 7f943cae3640 -1 rbd::mirror::InstanceWatcher: 0x562c1b562380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:49.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.097+0000 7f943d2e4640 -1 rbd::mirror::InstanceWatcher: 0x562c1b562a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:49.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.097+0000 7f9436ad7640 -1 rbd::mirror::MirrorStatusUpdater 0x562c1b5b4480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:49.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.097+0000 7f9436ad7640 -1 rbd::mirror::MirrorStatusUpdater 0x562c1b5b4360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:49.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.097+0000 7f94432f0640 -1 rbd::mirror::NamespaceReplayer: 0x562c1b5b2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:49.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.097+0000 7f94432f0640 -1 rbd::mirror::NamespaceReplayer: 0x562c1b5b21a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:49.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.098+0000 7f943cae3640 -1 rbd::mirror::MirrorStatusUpdater 0x562c1b5b4000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:49.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.098+0000 7f94432f0640 -1 rbd::mirror::NamespaceReplayer: 0x562c1b5b2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:49.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.098+0000 7f943d2e4640 -1 rbd::mirror::MirrorStatusUpdater 0x562c1b5b4120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:49.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.098+0000 7f94432f0640 -1 rbd::mirror::NamespaceReplayer: 0x562c1b5b21a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.131+0000 7f9436ad7640 -1 rbd::mirror::RemotePoolPoller: 0x562c1a3a05b0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-15T01:11:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.131+0000 7f94362d6640 -1 rbd::mirror::RemotePoolPoller: 0x562c1a3a05b0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-15T01:11:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.131+0000 7f94362d6640 -1 rbd::mirror::PoolReplayer: 0x562c1a3b3b00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-15T01:11:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:pool 'mirror' does not exist 2024-09-15T01:11:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T01:11:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-15T01:11:49.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.225+0000 7f9443af1640 -1 rbd::mirror::LeaderWatcher: 0x562c1b4c6c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:49.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.226+0000 7f9443af1640 -1 rbd::mirror::InstanceWatcher: 0x562c1b562000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5991 object: (2) No such file or directory 2024-09-15T01:11:49.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.226+0000 7f943cae3640 -1 rbd::mirror::InstanceWatcher: 0x562c1b562000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:49.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.227+0000 7f9436ad7640 -1 rbd::mirror::MirrorStatusUpdater 0x562c1b2a1200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:49.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.227+0000 7f94432f0640 -1 rbd::mirror::NamespaceReplayer: 0x562c1a3881a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:49.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.227+0000 7f943d2e4640 -1 rbd::mirror::MirrorStatusUpdater 0x562c1b2a17a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:49.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:49.227+0000 7f94432f0640 -1 rbd::mirror::NamespaceReplayer: 0x562c1a3881a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:pool 'mirror_parent' does not exist 2024-09-15T01:11:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ CEPH_ARGS= 2024-09-15T01:11:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-15T01:11:51.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.020+0000 7faed1c6f640 -1 rbd::mirror::RemotePoolPoller: 0x55725c024ea0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-15T01:11:51.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.020+0000 7faed2470640 -1 rbd::mirror::RemotePoolPoller: 0x55725c024ea0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-15T01:11:51.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.020+0000 7faed2470640 -1 rbd::mirror::PoolReplayer: 0x55725c037b00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-15T01:11:51.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.021+0000 7faedec89640 -1 rbd::mirror::InstanceWatcher: 0x55725d1cc380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6135 object: (2) No such file or directory 2024-09-15T01:11:51.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.021+0000 7faedec89640 -1 rbd::mirror::InstanceWatcher: 0x55725d1cc700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6135 object: (2) No such file or directory 2024-09-15T01:11:51.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.021+0000 7faed8c7d640 -1 rbd::mirror::InstanceWatcher: 0x55725d1cc380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.021+0000 7faed847c640 -1 rbd::mirror::InstanceWatcher: 0x55725d1cc700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.022+0000 7faed1c6f640 -1 rbd::mirror::MirrorStatusUpdater 0x55725d216480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.022+0000 7faed1c6f640 -1 rbd::mirror::MirrorStatusUpdater 0x55725d216360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.022+0000 7faedec89640 -1 rbd::mirror::NamespaceReplayer: 0x55725d214000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.022+0000 7faedec89640 -1 rbd::mirror::NamespaceReplayer: 0x55725d2141a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.022+0000 7faed8c7d640 -1 rbd::mirror::MirrorStatusUpdater 0x55725d216000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.022+0000 7faed8c7d640 -1 rbd::mirror::MirrorStatusUpdater 0x55725d216120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.022+0000 7faedf48a640 -1 rbd::mirror::NamespaceReplayer: 0x55725d214000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.022+0000 7faedf48a640 -1 rbd::mirror::NamespaceReplayer: 0x55725d2141a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.086+0000 7faec4454640 -1 rbd::mirror::RemotePoolPoller: 0x55725c025ba0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-15T01:11:51.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.086+0000 7faec4454640 -1 rbd::mirror::RemotePoolPoller: 0x55725c025ba0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-15T01:11:51.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.086+0000 7faec4454640 -1 rbd::mirror::PoolReplayer: 0x55725d12cf00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-15T01:11:51.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.105+0000 7faedf48a640 -1 rbd::mirror::LeaderWatcher: 0x55725d12cc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.105+0000 7faedf48a640 -1 rbd::mirror::InstanceWatcher: 0x55725d1cc000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6135 object: (2) No such file or directory 2024-09-15T01:11:51.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.106+0000 7faed847c640 -1 rbd::mirror::InstanceWatcher: 0x55725d1cc000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.106+0000 7faed2470640 -1 rbd::mirror::MirrorStatusUpdater 0x55725cf886c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.106+0000 7faedf48a640 -1 rbd::mirror::NamespaceReplayer: 0x55725da3e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.106+0000 7faed8c7d640 -1 rbd::mirror::MirrorStatusUpdater 0x55725cf370e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.106+0000 7faedec89640 -1 rbd::mirror::NamespaceReplayer: 0x55725da3e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:pool 'mirror_parent' does not exist 2024-09-15T01:11:51.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.109+0000 7faecac61640 -1 rbd::mirror::InstanceWatcher: 0x55725d688000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-15T01:11:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stop_mirrors cluster1 2024-09-15T01:11:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1 2024-09-15T01:11:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local sig= 2024-09-15T01:11:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 0 3 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stop_mirror cluster1:0 '' 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1:0 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local sig= 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stop_mirror cluster1:1 '' 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1:1 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local sig= 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T01:11:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T01:11:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:11:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stop_mirror cluster1:2 '' 2024-09-15T01:11:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1:2 2024-09-15T01:11:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local sig= 2024-09-15T01:11:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T01:11:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T01:11:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:11:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stop_mirror cluster1:3 '' 2024-09-15T01:11:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster1:3 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local sig= 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stop_mirrors cluster2 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local sig= 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:++ seq 0 3 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stop_mirror cluster2:0 '' 2024-09-15T01:11:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2:0 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local sig= 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stop_mirror cluster2:1 '' 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2:1 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local sig= 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stop_mirror cluster2:2 '' 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2:2 2024-09-15T01:11:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local sig= 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ stop_mirror cluster2:3 '' 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local cluster=cluster2:3 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ local sig= 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test -n 1 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ return 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' -z 1 ']' 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ test '' = /tmp/tmp.nS85ynkalX 2024-09-15T01:11:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ rm -Rf /tmp/tmp.nS85ynkalX 2024-09-15T01:11:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ '[' 0 -eq 0 ']' 2024-09-15T01:11:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ echo OK 2024-09-15T01:11:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stdout:OK 2024-09-15T01:11:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi059.stderr:+ exit 0 2024-09-15T01:11:51.142 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T01:11:51.142 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-15T01:11:51.245 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-15T01:11:51.245 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-15T01:11:51.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.341+0000 7f0d7c902640 -1 rbd::mirror::InstanceWatcher: 0x55ca80f23888 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.354+0000 7f0d83910640 -1 rbd::mirror::LeaderWatcher: 0x55ca8049ec00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-15T01:11:51.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.354+0000 7f0d7d103640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ca80957b00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-15T01:11:51.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.354+0000 7f0d7d103640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ca80957d40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-15T01:11:51.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.354+0000 7f0d7d103640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ca809578c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-15T01:11:51.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.354+0000 7f0d7c902640 -1 rbd::mirror::PoolWatcher: 0x55ca80ab03c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:51.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.354+0000 7f0d7c902640 -1 rbd::mirror::PoolWatcher: 0x55ca80f5e5a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:51.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.354+0000 7f0d7d103640 -1 rbd::mirror::PoolWatcher: 0x55ca80f5e1e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:51.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.354+0000 7f0d760f5640 -1 rbd::mirror::PoolWatcher: 0x55ca80690f00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:51.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.354+0000 7f0d760f5640 -1 rbd::mirror::PoolWatcher: 0x55ca80edbc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:51.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.354+0000 7f0d768f6640 -1 rbd::mirror::PoolWatcher: 0x55ca80f5e3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:51.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.355+0000 7f0d83910640 -1 rbd::mirror::LeaderWatcher: 0x55ca8049ec00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-15T01:11:51.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.355+0000 7f0d83910640 -1 rbd::mirror::LeaderWatcher: 0x55ca8049ec00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-15T01:11:51.520 DEBUG:teuthology.parallel:result is None 2024-09-15T01:11:51.520 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-15T01:11:51.546 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-15T01:11:51.546 DEBUG:teuthology.orchestra.run.smithi059:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-15T01:11:51.607 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-15T01:11:51.607 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-15T01:11:51.618 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-15T01:11:51.618 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 17 2024-09-15T01:11:51.619 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-15T01:11:51.619 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-15T01:11:51.663 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-15T01:11:51.663 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 16 2024-09-15T01:11:51.663 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-15T01:11:51.663 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-15T01:11:51.714 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-15T01:11:51.714 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:11:51.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.713+0000 7f3b1bd11640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 71609) UID: 1000 2024-09-15T01:11:51.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.714+0000 7f3b1ad0f640 -1 rbd::mirror::LeaderWatcher: 0x56219cb39200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.714+0000 7f3b1ad0f640 -1 rbd::mirror::InstanceWatcher: 0x56219c916700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6724 object: (2) No such file or directory 2024-09-15T01:11:51.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.714+0000 7f3b1ad0f640 -1 rbd::mirror::InstanceWatcher: 0x56219c916380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6724 object: (2) No such file or directory 2024-09-15T01:11:51.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.714+0000 7f3b1ad0f640 -1 rbd::mirror::InstanceWatcher: 0x56219cde6000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6727 object: (2) No such file or directory 2024-09-15T01:11:51.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.714+0000 7f3b14502640 -1 rbd::mirror::InstanceWatcher: 0x56219c916700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.714+0000 7f3b14502640 -1 rbd::mirror::InstanceWatcher: 0x56219c916380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.714+0000 7f3b074e8640 -1 rbd::mirror::InstanceWatcher: 0x56219cde6000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b0dcf5640 -1 rbd::mirror::MirrorStatusUpdater 0x56219c96a7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b0dcf5640 -1 rbd::mirror::MirrorStatusUpdater 0x56219c96a360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b1b510640 -1 rbd::mirror::NamespaceReplayer: 0x56219c968000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b1b510640 -1 rbd::mirror::NamespaceReplayer: 0x56219c9681a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b00cdb640 -1 rbd::mirror::MirrorStatusUpdater 0x56219cb747e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b1ad0f640 -1 rbd::mirror::NamespaceReplayer: 0x56219cde4000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b14d03640 -1 rbd::mirror::MirrorStatusUpdater 0x56219c96a000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b14d03640 -1 rbd::mirror::MirrorStatusUpdater 0x56219c96a120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b1b510640 -1 rbd::mirror::NamespaceReplayer: 0x56219c968000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b1b510640 -1 rbd::mirror::NamespaceReplayer: 0x56219c9681a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b06ce7640 -1 rbd::mirror::MirrorStatusUpdater 0x56219cb530e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.715+0000 7f3b1b510640 -1 rbd::mirror::NamespaceReplayer: 0x56219cde4000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.717+0000 7f3b1ad0f640 -1 rbd::mirror::LeaderWatcher: 0x56219c880c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.717+0000 7f3b1ad0f640 -1 rbd::mirror::InstanceWatcher: 0x56219c916000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6724 object: (2) No such file or directory 2024-09-15T01:11:51.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.717+0000 7f3b14502640 -1 rbd::mirror::InstanceWatcher: 0x56219c916000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.718+0000 7f3b0e4f6640 -1 rbd::mirror::MirrorStatusUpdater 0x56219c6ba6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.718+0000 7f3b1ad0f640 -1 rbd::mirror::NamespaceReplayer: 0x56219c59a1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.718+0000 7f3b14d03640 -1 rbd::mirror::MirrorStatusUpdater 0x56219c6b90e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.718+0000 7f3b1b510640 -1 rbd::mirror::NamespaceReplayer: 0x56219c59a1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-15T01:11:51.731 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-15T01:11:51.764 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-15T01:11:51.764 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:11:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.771+0000 7f91c8949640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 70145) UID: 1000 2024-09-15T01:11:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.772+0000 7f91c7947640 -1 rbd::mirror::LeaderWatcher: 0x55c02701b200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.772+0000 7f91c8148640 -1 rbd::mirror::InstanceWatcher: 0x55c026db2a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6623 object: (2) No such file or directory 2024-09-15T01:11:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.772+0000 7f91c8148640 -1 rbd::mirror::InstanceWatcher: 0x55c026db2380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6623 object: (2) No such file or directory 2024-09-15T01:11:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.772+0000 7f91c8148640 -1 rbd::mirror::InstanceWatcher: 0x55c02728e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6626 object: (2) No such file or directory 2024-09-15T01:11:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.772+0000 7f91c113a640 -1 rbd::mirror::InstanceWatcher: 0x55c026db2a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.772+0000 7f91c193b640 -1 rbd::mirror::InstanceWatcher: 0x55c026db2380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91b4120640 -1 rbd::mirror::InstanceWatcher: 0x55c02728e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91bb12e640 -1 rbd::mirror::MirrorStatusUpdater 0x55c026e20b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91bb12e640 -1 rbd::mirror::MirrorStatusUpdater 0x55c026e205a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91c8148640 -1 rbd::mirror::NamespaceReplayer: 0x55c026daa340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91c8148640 -1 rbd::mirror::NamespaceReplayer: 0x55c026daa4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91ad913640 -1 rbd::mirror::MirrorStatusUpdater 0x55c0270407e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91c7947640 -1 rbd::mirror::NamespaceReplayer: 0x55c026daa1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91c113a640 -1 rbd::mirror::MirrorStatusUpdater 0x55c026e20240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91c113a640 -1 rbd::mirror::MirrorStatusUpdater 0x55c026e20360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91c8148640 -1 rbd::mirror::NamespaceReplayer: 0x55c026daa340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91c8148640 -1 rbd::mirror::NamespaceReplayer: 0x55c026daa4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91b391f640 -1 rbd::mirror::MirrorStatusUpdater 0x55c026fd9200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.773+0000 7f91c7947640 -1 rbd::mirror::NamespaceReplayer: 0x55c026daa1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.775+0000 7f91c8148640 -1 rbd::mirror::LeaderWatcher: 0x55c026d1cc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.776+0000 7f91c8148640 -1 rbd::mirror::InstanceWatcher: 0x55c026db2000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6623 object: (2) No such file or directory 2024-09-15T01:11:51.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.776+0000 7f91c193b640 -1 rbd::mirror::InstanceWatcher: 0x55c026db2000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.776+0000 7f91ba92d640 -1 rbd::mirror::MirrorStatusUpdater 0x55c026b7e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.776+0000 7f91c7947640 -1 rbd::mirror::NamespaceReplayer: 0x55c026daa000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.776+0000 7f91c113a640 -1 rbd::mirror::MirrorStatusUpdater 0x55c026ad30e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi059.stderr:2024-09-15T01:11:51.776+0000 7f91c8148640 -1 rbd::mirror::NamespaceReplayer: 0x55c026daa000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-15T01:11:51.790 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-15T01:11:51.817 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-15T01:11:51.817 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:11:51.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.830+0000 7f0d84111640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 55651) UID: 1000 2024-09-15T01:11:51.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.831+0000 7f0d6f0e7640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ca80684000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-15T01:11:51.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.831+0000 7f0d83910640 -1 rbd::mirror::InstanceWatcher: 0x55ca80542380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5660 object: (2) No such file or directory 2024-09-15T01:11:51.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.831+0000 7f0d83910640 -1 rbd::mirror::InstanceWatcher: 0x55ca80542700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5660 object: (2) No such file or directory 2024-09-15T01:11:51.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.831+0000 7f0d6f0e7640 -1 rbd::mirror::PoolWatcher: 0x55ca80ab0000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:51.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.831+0000 7f0d688da640 -1 rbd::mirror::PoolWatcher: 0x55ca80af9e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:51.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.831+0000 7f0d7d103640 -1 rbd::mirror::InstanceWatcher: 0x55ca80542380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.831+0000 7f0d7c902640 -1 rbd::mirror::InstanceWatcher: 0x55ca80542700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.831+0000 7f0d760f5640 -1 rbd::mirror::MirrorStatusUpdater 0x55ca8059c5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.831+0000 7f0d760f5640 -1 rbd::mirror::MirrorStatusUpdater 0x55ca8059c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.831+0000 7f0d8310f640 -1 rbd::mirror::NamespaceReplayer: 0x55ca8059a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.832+0000 7f0d83910640 -1 rbd::mirror::NamespaceReplayer: 0x55ca8059a1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.832+0000 7f0d83910640 -1 rbd::mirror::LeaderWatcher: 0x55ca8074f200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-15T01:11:51.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.832+0000 7f0d83910640 -1 rbd::mirror::LeaderWatcher: 0x55ca8074f200 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-15T01:11:51.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.832+0000 7f0d7d103640 -1 rbd::mirror::MirrorStatusUpdater 0x55ca8059c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.832+0000 7f0d8310f640 -1 rbd::mirror::NamespaceReplayer: 0x55ca8059a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.832+0000 7f0d7c902640 -1 rbd::mirror::MirrorStatusUpdater 0x55ca8059c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.832+0000 7f0d83910640 -1 rbd::mirror::NamespaceReplayer: 0x55ca8059a1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.832+0000 7f0d83910640 -1 rbd::mirror::LeaderWatcher: 0x55ca8074f200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.832+0000 7f0d83910640 -1 rbd::mirror::InstanceWatcher: 0x55ca80a0a000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5662 object: (2) No such file or directory 2024-09-15T01:11:51.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.832+0000 7f0d6f0e7640 -1 rbd::mirror::InstanceWatcher: 0x55ca80a0a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.833+0000 7f0d688da640 -1 rbd::mirror::MirrorStatusUpdater 0x55ca8059c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.833+0000 7f0d8310f640 -1 rbd::mirror::NamespaceReplayer: 0x55ca802141a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.833+0000 7f0d6f8e8640 -1 rbd::mirror::MirrorStatusUpdater 0x55ca8077d440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.833+0000 7f0d83910640 -1 rbd::mirror::NamespaceReplayer: 0x55ca802141a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.835+0000 7f0d83910640 -1 rbd::mirror::LeaderWatcher: 0x55ca8049ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.835+0000 7f0d83910640 -1 rbd::mirror::InstanceWatcher: 0x55ca80542000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5660 object: (2) No such file or directory 2024-09-15T01:11:51.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.836+0000 7f0d7d103640 -1 rbd::mirror::InstanceWatcher: 0x55ca80542000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.836+0000 7f0d768f6640 -1 rbd::mirror::MirrorStatusUpdater 0x55ca803125a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.836+0000 7f0d8310f640 -1 rbd::mirror::NamespaceReplayer: 0x55ca80214000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.836+0000 7f0d7c902640 -1 rbd::mirror::MirrorStatusUpdater 0x55ca802b70e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi059.stderr:2024-09-15T01:11:51.836+0000 7f0d83910640 -1 rbd::mirror::NamespaceReplayer: 0x55ca80214000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-15T01:11:51.850 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-15T01:11:51.866 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-15T01:11:51.866 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:11:51.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.890+0000 7f7ca7652640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 70767) UID: 1000 2024-09-15T01:11:51.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.891+0000 7f7ca6650640 -1 rbd::mirror::LeaderWatcher: 0x5640107ab200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.891+0000 7f7ca6650640 -1 rbd::mirror::InstanceWatcher: 0x56401055e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6674 object: (2) No such file or directory 2024-09-15T01:11:51.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.891+0000 7f7ca6650640 -1 rbd::mirror::InstanceWatcher: 0x56401055e700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6674 object: (2) No such file or directory 2024-09-15T01:11:51.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.891+0000 7f7ca6650640 -1 rbd::mirror::InstanceWatcher: 0x56401055ea80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6676 object: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.891+0000 7f7ca0644640 -1 rbd::mirror::InstanceWatcher: 0x56401055e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.891+0000 7f7ca0644640 -1 rbd::mirror::InstanceWatcher: 0x56401055e700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.891+0000 7f7c92628640 -1 rbd::mirror::InstanceWatcher: 0x56401055ea80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7c99636640 -1 rbd::mirror::MirrorStatusUpdater 0x5640105cc360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7ca6650640 -1 rbd::mirror::NamespaceReplayer: 0x5640105ca000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7c99e37640 -1 rbd::mirror::MirrorStatusUpdater 0x5640105cc5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7ca6e51640 -1 rbd::mirror::NamespaceReplayer: 0x5640105ca1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7c8be1b640 -1 rbd::mirror::MirrorStatusUpdater 0x5640105cc6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7ca6650640 -1 rbd::mirror::NamespaceReplayer: 0x5640105ca4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7c9fe43640 -1 rbd::mirror::MirrorStatusUpdater 0x5640105cc000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7ca6650640 -1 rbd::mirror::NamespaceReplayer: 0x5640105ca000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7ca0644640 -1 rbd::mirror::MirrorStatusUpdater 0x5640105cc120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7ca6650640 -1 rbd::mirror::NamespaceReplayer: 0x5640105ca1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7c92e29640 -1 rbd::mirror::MirrorStatusUpdater 0x5640107970e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.892+0000 7f7ca6e51640 -1 rbd::mirror::NamespaceReplayer: 0x5640105ca4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.894+0000 7f7ca6e51640 -1 rbd::mirror::LeaderWatcher: 0x5640104d0c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.894+0000 7f7ca6e51640 -1 rbd::mirror::InstanceWatcher: 0x56401055e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6674 object: (2) No such file or directory 2024-09-15T01:11:51.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.895+0000 7f7c9fe43640 -1 rbd::mirror::InstanceWatcher: 0x56401055e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.895+0000 7f7c99636640 -1 rbd::mirror::MirrorStatusUpdater 0x56401030e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.895+0000 7f7ca6650640 -1 rbd::mirror::NamespaceReplayer: 0x564010dba000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.895+0000 7f7ca0644640 -1 rbd::mirror::MirrorStatusUpdater 0x5640102b6fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi059.stderr:2024-09-15T01:11:51.895+0000 7f7ca6e51640 -1 rbd::mirror::NamespaceReplayer: 0x564010dba000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-15T01:11:51.908 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-15T01:11:51.927 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-15T01:11:51.928 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:11:51.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.948+0000 7f85ad85d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 36677) UID: 1000 2024-09-15T01:11:51.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.949+0000 7f8599034640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f9d5858480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-15T01:11:51.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.949+0000 7f8598833640 -1 rbd::mirror::PoolWatcher: 0x55f9d5aff860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.949+0000 7f85ad05c640 -1 rbd::mirror::InstanceWatcher: 0x55f9d52c7180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4295 object: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.949+0000 7f85ad05c640 -1 rbd::mirror::InstanceWatcher: 0x55f9d58f6000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4295 object: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.949+0000 7f8592026640 -1 rbd::mirror::PoolWatcher: 0x55f9d43634a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.949+0000 7f85a684f640 -1 rbd::mirror::InstanceWatcher: 0x55f9d52c7180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.949+0000 7f85a684f640 -1 rbd::mirror::InstanceWatcher: 0x55f9d58f6000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.949+0000 7f85ac85b640 -1 rbd::mirror::LeaderWatcher: 0x55f9d5a5ec00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.949+0000 7f85ac85b640 -1 rbd::mirror::LeaderWatcher: 0x55f9d5a5ec00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f859f841640 -1 rbd::mirror::MirrorStatusUpdater 0x55f9d5648360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f859f841640 -1 rbd::mirror::MirrorStatusUpdater 0x55f9d56485a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f85ad05c640 -1 rbd::mirror::NamespaceReplayer: 0x55f9d5646000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f85ad05c640 -1 rbd::mirror::NamespaceReplayer: 0x55f9d56461a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f85ad05c640 -1 rbd::mirror::LeaderWatcher: 0x55f9d5a5ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f85a684f640 -1 rbd::mirror::MirrorStatusUpdater 0x55f9d5648000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f85a684f640 -1 rbd::mirror::MirrorStatusUpdater 0x55f9d5648120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f85ac85b640 -1 rbd::mirror::NamespaceReplayer: 0x55f9d5646000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f85ac85b640 -1 rbd::mirror::NamespaceReplayer: 0x55f9d56461a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f85ac85b640 -1 rbd::mirror::InstanceWatcher: 0x55f9d58f7c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4298 object: (2) No such file or directory 2024-09-15T01:11:51.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.950+0000 7f8598833640 -1 rbd::mirror::InstanceWatcher: 0x55f9d58f7c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.951+0000 7f8592827640 -1 rbd::mirror::MirrorStatusUpdater 0x55f9d58b0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.951+0000 7f85ac85b640 -1 rbd::mirror::NamespaceReplayer: 0x55f9d5102340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.951+0000 7f8599034640 -1 rbd::mirror::MirrorStatusUpdater 0x55f9d587d0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.951+0000 7f85ad05c640 -1 rbd::mirror::NamespaceReplayer: 0x55f9d5102340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.953+0000 7f85ad05c640 -1 rbd::mirror::LeaderWatcher: 0x55f9d5331800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:51.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.954+0000 7f85ad05c640 -1 rbd::mirror::InstanceWatcher: 0x55f9d52c6e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4295 object: (2) No such file or directory 2024-09-15T01:11:51.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.954+0000 7f85a604e640 -1 rbd::mirror::InstanceWatcher: 0x55f9d52c6e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:51.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.954+0000 7f85a0042640 -1 rbd::mirror::MirrorStatusUpdater 0x55f9d538a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.954+0000 7f85ad05c640 -1 rbd::mirror::NamespaceReplayer: 0x55f9d5496000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.954+0000 7f85a684f640 -1 rbd::mirror::MirrorStatusUpdater 0x55f9d530cea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:51.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi059.stderr:2024-09-15T01:11:51.954+0000 7f85ac85b640 -1 rbd::mirror::NamespaceReplayer: 0x55f9d5496000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:51.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-15T01:11:51.969 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-15T01:11:51.983 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-15T01:11:51.983 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:11:52.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:52.008+0000 7f94442f2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 68368) UID: 1000 2024-09-15T01:11:52.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:52.009+0000 7f94432f0640 -1 rbd::mirror::LeaderWatcher: 0x562c1b787200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:52.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:52.009+0000 7f94432f0640 -1 rbd::mirror::InstanceWatcher: 0x562c1b562700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5993 object: (2) No such file or directory 2024-09-15T01:11:52.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:52.009+0000 7f942f2c8640 -1 rbd::mirror::InstanceWatcher: 0x562c1b562700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:52.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:52.010+0000 7f9428abb640 -1 rbd::mirror::MirrorStatusUpdater 0x562c1b5b4b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:52.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:52.010+0000 7f9443af1640 -1 rbd::mirror::NamespaceReplayer: 0x562c1b5b24e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:52.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:52.010+0000 7f942fac9640 -1 rbd::mirror::MirrorStatusUpdater 0x562c1b793560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:52.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi059.stderr:2024-09-15T01:11:52.010+0000 7f94432f0640 -1 rbd::mirror::NamespaceReplayer: 0x562c1b5b24e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:52.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-15T01:11:52.023 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-15T01:11:52.034 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-15T01:11:52.034 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:11:52.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.063+0000 7fb2995de640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 69514) UID: 1000 2024-09-15T01:11:52.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.064+0000 7fb298ddd640 -1 rbd::mirror::InstanceWatcher: 0x562684620380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6039 object: (2) No such file or directory 2024-09-15T01:11:52.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.064+0000 7fb2985dc640 -1 rbd::mirror::InstanceWatcher: 0x562684620700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6039 object: (2) No such file or directory 2024-09-15T01:11:52.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.064+0000 7fb2845b4640 -1 rbd::mirror::InstanceWatcher: 0x562684620a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:52.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.065+0000 7fb2925d0640 -1 rbd::mirror::InstanceWatcher: 0x562684620380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:52.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.065+0000 7fb291dcf640 -1 rbd::mirror::InstanceWatcher: 0x562684620700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:52.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.065+0000 7fb28b5c2640 -1 rbd::mirror::MirrorStatusUpdater 0x562684674360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:52.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.065+0000 7fb28b5c2640 -1 rbd::mirror::MirrorStatusUpdater 0x5626846745a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:52.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.065+0000 7fb2985dc640 -1 rbd::mirror::NamespaceReplayer: 0x562684672000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:52.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.065+0000 7fb2985dc640 -1 rbd::mirror::NamespaceReplayer: 0x5626846721a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:52.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.065+0000 7fb291dcf640 -1 rbd::mirror::MirrorStatusUpdater 0x562684674000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:52.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.065+0000 7fb291dcf640 -1 rbd::mirror::MirrorStatusUpdater 0x562684674120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:52.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.065+0000 7fb298ddd640 -1 rbd::mirror::NamespaceReplayer: 0x562684672000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:52.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.065+0000 7fb298ddd640 -1 rbd::mirror::NamespaceReplayer: 0x5626846721a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:52.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.067+0000 7fb298ddd640 -1 rbd::mirror::LeaderWatcher: 0x56268458ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-15T01:11:52.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.069+0000 7fb298ddd640 -1 rbd::mirror::InstanceWatcher: 0x562684620000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6039 object: (2) No such file or directory 2024-09-15T01:11:52.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.069+0000 7fb2925d0640 -1 rbd::mirror::InstanceWatcher: 0x562684620000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-15T01:11:52.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.069+0000 7fb28bdc3640 -1 rbd::mirror::MirrorStatusUpdater 0x5626844085a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:52.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.069+0000 7fb2985dc640 -1 rbd::mirror::NamespaceReplayer: 0x56268346c680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-15T01:11:52.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.069+0000 7fb291dcf640 -1 rbd::mirror::MirrorStatusUpdater 0x5626843a7200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-15T01:11:52.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi059.stderr:2024-09-15T01:11:52.069+0000 7fb298ddd640 -1 rbd::mirror::NamespaceReplayer: 0x56268346c680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-15T01:11:52.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-15T01:11:52.082 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-15T01:11:52.095 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-15T01:11:52.095 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:11:52.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi059.stderr:2024-09-15T01:11:52.122+0000 7faedfc8b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 71499) UID: 1000 2024-09-15T01:11:52.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-15T01:11:52.133 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T01:11:52.143 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-15T01:11:52.143 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-15T01:11:52.473 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T01:11:52.474 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-15T01:11:52.486 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":1608,"stamp":"2024-09-15T01:11:50.994808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4534040,"kb_used_data":17696,"kb_used_omap":4,"kb_used_meta":4516283,"kb_avail":278581480,"statfs":{"total":289910292480,"available":285267435520,"internally_reserved":0,"allocated":18120704,"data_stored":10157624,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4624674127},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536617198,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-136246,"num_read_kb":-39664664,"num_write":-647701,"num_write_kb":-236761031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-24755248,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.012859"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.138634+0000","last_change":"2024-09-15T00:35:13.131214+0000","last_active":"2024-09-15T01:11:49.138634+0000","last_peered":"2024-09-15T01:11:49.138634+0000","last_clean":"2024-09-15T01:11:49.138634+0000","last_became_active":"2024-09-15T00:35:11.119667+0000","last_became_peered":"2024-09-15T00:35:11.119667+0000","last_unstale":"2024-09-15T01:11:49.138634+0000","last_undegraded":"2024-09-15T01:11:49.138634+0000","last_fullsized":"2024-09-15T01:11:49.138634+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:56:15.752164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125682,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.145134+0000","last_change":"2024-09-15T00:35:13.131275+0000","last_active":"2024-09-15T01:11:49.145134+0000","last_peered":"2024-09-15T01:11:49.145134+0000","last_clean":"2024-09-15T01:11:49.145134+0000","last_became_active":"2024-09-15T00:35:11.119728+0000","last_became_peered":"2024-09-15T00:35:11.119728+0000","last_unstale":"2024-09-15T01:11:49.145134+0000","last_undegraded":"2024-09-15T01:11:49.145134+0000","last_fullsized":"2024-09-15T01:11:49.145134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:38:32.666290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193798,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.138361+0000","last_change":"2024-09-15T00:35:13.131573+0000","last_active":"2024-09-15T01:11:49.138361+0000","last_peered":"2024-09-15T01:11:49.138361+0000","last_clean":"2024-09-15T01:11:49.138361+0000","last_became_active":"2024-09-15T00:35:11.120732+0000","last_became_peered":"2024-09-15T00:35:11.120732+0000","last_unstale":"2024-09-15T01:11:49.138361+0000","last_undegraded":"2024-09-15T01:11:49.138361+0000","last_fullsized":"2024-09-15T01:11:49.138361+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:01:09.277084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001482562,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.142798+0000","last_change":"2024-09-15T00:35:13.130700+0000","last_active":"2024-09-15T01:11:49.142798+0000","last_peered":"2024-09-15T01:11:49.142798+0000","last_clean":"2024-09-15T01:11:49.142798+0000","last_became_active":"2024-09-15T00:35:11.120070+0000","last_became_peered":"2024-09-15T00:35:11.120070+0000","last_unstale":"2024-09-15T01:11:49.142798+0000","last_undegraded":"2024-09-15T01:11:49.142798+0000","last_fullsized":"2024-09-15T01:11:49.142798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:26:49.101024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070255600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1016,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-15T01:11:42.989247+0000","last_change":"2024-09-15T00:35:13.128765+0000","last_active":"2024-09-15T01:11:42.989247+0000","last_peered":"2024-09-15T01:11:42.989247+0000","last_clean":"2024-09-15T01:11:42.989247+0000","last_became_active":"2024-09-15T00:35:11.120934+0000","last_became_peered":"2024-09-15T00:35:11.120934+0000","last_unstale":"2024-09-15T01:11:42.989247+0000","last_undegraded":"2024-09-15T01:11:42.989247+0000","last_fullsized":"2024-09-15T01:11:42.989247+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:45:31.683827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011530049999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.006342+0000","last_change":"2024-09-15T00:35:13.345445+0000","last_active":"2024-09-15T01:11:49.006342+0000","last_peered":"2024-09-15T01:11:49.006342+0000","last_clean":"2024-09-15T01:11:49.006342+0000","last_became_active":"2024-09-15T00:35:11.123441+0000","last_became_peered":"2024-09-15T00:35:11.123441+0000","last_unstale":"2024-09-15T01:11:49.006342+0000","last_undegraded":"2024-09-15T01:11:49.006342+0000","last_fullsized":"2024-09-15T01:11:49.006342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:34:52.254834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000350637,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.005803+0000","last_change":"2024-09-15T00:35:13.345476+0000","last_active":"2024-09-15T01:11:49.005803+0000","last_peered":"2024-09-15T01:11:49.005803+0000","last_clean":"2024-09-15T01:11:49.005803+0000","last_became_active":"2024-09-15T00:35:11.123456+0000","last_became_peered":"2024-09-15T00:35:11.123456+0000","last_unstale":"2024-09-15T01:11:49.005803+0000","last_undegraded":"2024-09-15T01:11:49.005803+0000","last_fullsized":"2024-09-15T01:11:49.005803+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:43.436651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038329899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.137518+0000","last_change":"2024-09-15T00:35:13.130729+0000","last_active":"2024-09-15T01:11:49.137518+0000","last_peered":"2024-09-15T01:11:49.137518+0000","last_clean":"2024-09-15T01:11:49.137518+0000","last_became_active":"2024-09-15T00:35:11.119438+0000","last_became_peered":"2024-09-15T00:35:11.119438+0000","last_unstale":"2024-09-15T01:11:49.137518+0000","last_undegraded":"2024-09-15T01:11:49.137518+0000","last_fullsized":"2024-09-15T01:11:49.137518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:02:21.034948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071155299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1143,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.136645+0000","last_change":"2024-09-15T00:35:06.101340+0000","last_active":"2024-09-15T01:11:49.136645+0000","last_peered":"2024-09-15T01:11:49.136645+0000","last_clean":"2024-09-15T01:11:49.136645+0000","last_became_active":"2024-09-15T00:35:06.100362+0000","last_became_peered":"2024-09-15T00:35:06.100362+0000","last_unstale":"2024-09-15T01:11:49.136645+0000","last_undegraded":"2024-09-15T01:11:49.136645+0000","last_fullsized":"2024-09-15T01:11:49.136645+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_clean_scrub_stamp":"2024-09-15T00:35:05.089463+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:04.296940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706111,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1454268,"kb_used_data":5864,"kb_used_omap":1,"kb_used_meta":1448382,"kb_avail":92917572,"statfs":{"total":96636764160,"available":95147593728,"internally_reserved":0,"allocated":6004736,"data_stored":2994664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1483143621},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1476344,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":1470398,"kb_avail":92895496,"statfs":{"total":96636764160,"available":95124987904,"internally_reserved":0,"allocated":6066176,"data_stored":3585051,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1505688005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1603428,"kb_used_data":5908,"kb_used_omap":1,"kb_used_meta":1597502,"kb_avail":92768412,"statfs":{"total":96636764160,"available":94994853888,"internally_reserved":0,"allocated":6049792,"data_stored":3577909,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1635842501},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:11:52.487 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-15T01:11:52.774 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T01:11:52.775 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-15T01:11:52.788 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":1609,"stamp":"2024-09-15T01:11:52.670461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4534040,"kb_used_data":17696,"kb_used_omap":4,"kb_used_meta":4516283,"kb_avail":278581480,"statfs":{"total":289910292480,"available":285267435520,"internally_reserved":0,"allocated":18120704,"data_stored":10157624,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4624674127},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536617198,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-136246,"num_read_kb":-39664664,"num_write":-647701,"num_write_kb":-236761031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-24755248,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002147"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.138634+0000","last_change":"2024-09-15T00:35:13.131214+0000","last_active":"2024-09-15T01:11:49.138634+0000","last_peered":"2024-09-15T01:11:49.138634+0000","last_clean":"2024-09-15T01:11:49.138634+0000","last_became_active":"2024-09-15T00:35:11.119667+0000","last_became_peered":"2024-09-15T00:35:11.119667+0000","last_unstale":"2024-09-15T01:11:49.138634+0000","last_undegraded":"2024-09-15T01:11:49.138634+0000","last_fullsized":"2024-09-15T01:11:49.138634+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:56:15.752164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125682,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.145134+0000","last_change":"2024-09-15T00:35:13.131275+0000","last_active":"2024-09-15T01:11:49.145134+0000","last_peered":"2024-09-15T01:11:49.145134+0000","last_clean":"2024-09-15T01:11:49.145134+0000","last_became_active":"2024-09-15T00:35:11.119728+0000","last_became_peered":"2024-09-15T00:35:11.119728+0000","last_unstale":"2024-09-15T01:11:49.145134+0000","last_undegraded":"2024-09-15T01:11:49.145134+0000","last_fullsized":"2024-09-15T01:11:49.145134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:38:32.666290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193798,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.138361+0000","last_change":"2024-09-15T00:35:13.131573+0000","last_active":"2024-09-15T01:11:49.138361+0000","last_peered":"2024-09-15T01:11:49.138361+0000","last_clean":"2024-09-15T01:11:49.138361+0000","last_became_active":"2024-09-15T00:35:11.120732+0000","last_became_peered":"2024-09-15T00:35:11.120732+0000","last_unstale":"2024-09-15T01:11:49.138361+0000","last_undegraded":"2024-09-15T01:11:49.138361+0000","last_fullsized":"2024-09-15T01:11:49.138361+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:01:09.277084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001482562,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.142798+0000","last_change":"2024-09-15T00:35:13.130700+0000","last_active":"2024-09-15T01:11:49.142798+0000","last_peered":"2024-09-15T01:11:49.142798+0000","last_clean":"2024-09-15T01:11:49.142798+0000","last_became_active":"2024-09-15T00:35:11.120070+0000","last_became_peered":"2024-09-15T00:35:11.120070+0000","last_unstale":"2024-09-15T01:11:49.142798+0000","last_undegraded":"2024-09-15T01:11:49.142798+0000","last_fullsized":"2024-09-15T01:11:49.142798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:26:49.101024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070255600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1016,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-15T01:11:42.989247+0000","last_change":"2024-09-15T00:35:13.128765+0000","last_active":"2024-09-15T01:11:42.989247+0000","last_peered":"2024-09-15T01:11:42.989247+0000","last_clean":"2024-09-15T01:11:42.989247+0000","last_became_active":"2024-09-15T00:35:11.120934+0000","last_became_peered":"2024-09-15T00:35:11.120934+0000","last_unstale":"2024-09-15T01:11:42.989247+0000","last_undegraded":"2024-09-15T01:11:42.989247+0000","last_fullsized":"2024-09-15T01:11:42.989247+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:45:31.683827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011530049999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.006342+0000","last_change":"2024-09-15T00:35:13.345445+0000","last_active":"2024-09-15T01:11:49.006342+0000","last_peered":"2024-09-15T01:11:49.006342+0000","last_clean":"2024-09-15T01:11:49.006342+0000","last_became_active":"2024-09-15T00:35:11.123441+0000","last_became_peered":"2024-09-15T00:35:11.123441+0000","last_unstale":"2024-09-15T01:11:49.006342+0000","last_undegraded":"2024-09-15T01:11:49.006342+0000","last_fullsized":"2024-09-15T01:11:49.006342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:34:52.254834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000350637,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.005803+0000","last_change":"2024-09-15T00:35:13.345476+0000","last_active":"2024-09-15T01:11:49.005803+0000","last_peered":"2024-09-15T01:11:49.005803+0000","last_clean":"2024-09-15T01:11:49.005803+0000","last_became_active":"2024-09-15T00:35:11.123456+0000","last_became_peered":"2024-09-15T00:35:11.123456+0000","last_unstale":"2024-09-15T01:11:49.005803+0000","last_undegraded":"2024-09-15T01:11:49.005803+0000","last_fullsized":"2024-09-15T01:11:49.005803+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:43.436651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038329899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.137518+0000","last_change":"2024-09-15T00:35:13.130729+0000","last_active":"2024-09-15T01:11:49.137518+0000","last_peered":"2024-09-15T01:11:49.137518+0000","last_clean":"2024-09-15T01:11:49.137518+0000","last_became_active":"2024-09-15T00:35:11.119438+0000","last_became_peered":"2024-09-15T00:35:11.119438+0000","last_unstale":"2024-09-15T01:11:49.137518+0000","last_undegraded":"2024-09-15T01:11:49.137518+0000","last_fullsized":"2024-09-15T01:11:49.137518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:02:21.034948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071155299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1143,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.136645+0000","last_change":"2024-09-15T00:35:06.101340+0000","last_active":"2024-09-15T01:11:49.136645+0000","last_peered":"2024-09-15T01:11:49.136645+0000","last_clean":"2024-09-15T01:11:49.136645+0000","last_became_active":"2024-09-15T00:35:06.100362+0000","last_became_peered":"2024-09-15T00:35:06.100362+0000","last_unstale":"2024-09-15T01:11:49.136645+0000","last_undegraded":"2024-09-15T01:11:49.136645+0000","last_fullsized":"2024-09-15T01:11:49.136645+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_clean_scrub_stamp":"2024-09-15T00:35:05.089463+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:04.296940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706111,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1454268,"kb_used_data":5864,"kb_used_omap":1,"kb_used_meta":1448382,"kb_avail":92917572,"statfs":{"total":96636764160,"available":95147593728,"internally_reserved":0,"allocated":6004736,"data_stored":2994664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1483143621},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1476344,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":1470398,"kb_avail":92895496,"statfs":{"total":96636764160,"available":95124987904,"internally_reserved":0,"allocated":6066176,"data_stored":3585051,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1505688005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1603428,"kb_used_data":5908,"kb_used_omap":1,"kb_used_meta":1597502,"kb_avail":92768412,"statfs":{"total":96636764160,"available":94994853888,"internally_reserved":0,"allocated":6049792,"data_stored":3577909,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1635842501},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:11:52.789 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-15T01:11:52.790 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-15T01:11:53.076 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T01:11:53.077 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-15T01:11:53.088 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":1609,"stamp":"2024-09-15T01:11:52.670461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4534040,"kb_used_data":17696,"kb_used_omap":4,"kb_used_meta":4516283,"kb_avail":278581480,"statfs":{"total":289910292480,"available":285267435520,"internally_reserved":0,"allocated":18120704,"data_stored":10157624,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4624674127},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536617198,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-136246,"num_read_kb":-39664664,"num_write":-647701,"num_write_kb":-236761031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-24755248,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002147"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.138634+0000","last_change":"2024-09-15T00:35:13.131214+0000","last_active":"2024-09-15T01:11:49.138634+0000","last_peered":"2024-09-15T01:11:49.138634+0000","last_clean":"2024-09-15T01:11:49.138634+0000","last_became_active":"2024-09-15T00:35:11.119667+0000","last_became_peered":"2024-09-15T00:35:11.119667+0000","last_unstale":"2024-09-15T01:11:49.138634+0000","last_undegraded":"2024-09-15T01:11:49.138634+0000","last_fullsized":"2024-09-15T01:11:49.138634+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:56:15.752164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125682,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.145134+0000","last_change":"2024-09-15T00:35:13.131275+0000","last_active":"2024-09-15T01:11:49.145134+0000","last_peered":"2024-09-15T01:11:49.145134+0000","last_clean":"2024-09-15T01:11:49.145134+0000","last_became_active":"2024-09-15T00:35:11.119728+0000","last_became_peered":"2024-09-15T00:35:11.119728+0000","last_unstale":"2024-09-15T01:11:49.145134+0000","last_undegraded":"2024-09-15T01:11:49.145134+0000","last_fullsized":"2024-09-15T01:11:49.145134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:38:32.666290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193798,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.138361+0000","last_change":"2024-09-15T00:35:13.131573+0000","last_active":"2024-09-15T01:11:49.138361+0000","last_peered":"2024-09-15T01:11:49.138361+0000","last_clean":"2024-09-15T01:11:49.138361+0000","last_became_active":"2024-09-15T00:35:11.120732+0000","last_became_peered":"2024-09-15T00:35:11.120732+0000","last_unstale":"2024-09-15T01:11:49.138361+0000","last_undegraded":"2024-09-15T01:11:49.138361+0000","last_fullsized":"2024-09-15T01:11:49.138361+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:01:09.277084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001482562,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.142798+0000","last_change":"2024-09-15T00:35:13.130700+0000","last_active":"2024-09-15T01:11:49.142798+0000","last_peered":"2024-09-15T01:11:49.142798+0000","last_clean":"2024-09-15T01:11:49.142798+0000","last_became_active":"2024-09-15T00:35:11.120070+0000","last_became_peered":"2024-09-15T00:35:11.120070+0000","last_unstale":"2024-09-15T01:11:49.142798+0000","last_undegraded":"2024-09-15T01:11:49.142798+0000","last_fullsized":"2024-09-15T01:11:49.142798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:26:49.101024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070255600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1016,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-15T01:11:42.989247+0000","last_change":"2024-09-15T00:35:13.128765+0000","last_active":"2024-09-15T01:11:42.989247+0000","last_peered":"2024-09-15T01:11:42.989247+0000","last_clean":"2024-09-15T01:11:42.989247+0000","last_became_active":"2024-09-15T00:35:11.120934+0000","last_became_peered":"2024-09-15T00:35:11.120934+0000","last_unstale":"2024-09-15T01:11:42.989247+0000","last_undegraded":"2024-09-15T01:11:42.989247+0000","last_fullsized":"2024-09-15T01:11:42.989247+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:45:31.683827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011530049999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.006342+0000","last_change":"2024-09-15T00:35:13.345445+0000","last_active":"2024-09-15T01:11:49.006342+0000","last_peered":"2024-09-15T01:11:49.006342+0000","last_clean":"2024-09-15T01:11:49.006342+0000","last_became_active":"2024-09-15T00:35:11.123441+0000","last_became_peered":"2024-09-15T00:35:11.123441+0000","last_unstale":"2024-09-15T01:11:49.006342+0000","last_undegraded":"2024-09-15T01:11:49.006342+0000","last_fullsized":"2024-09-15T01:11:49.006342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:34:52.254834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000350637,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.005803+0000","last_change":"2024-09-15T00:35:13.345476+0000","last_active":"2024-09-15T01:11:49.005803+0000","last_peered":"2024-09-15T01:11:49.005803+0000","last_clean":"2024-09-15T01:11:49.005803+0000","last_became_active":"2024-09-15T00:35:11.123456+0000","last_became_peered":"2024-09-15T00:35:11.123456+0000","last_unstale":"2024-09-15T01:11:49.005803+0000","last_undegraded":"2024-09-15T01:11:49.005803+0000","last_fullsized":"2024-09-15T01:11:49.005803+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:43.436651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038329899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.137518+0000","last_change":"2024-09-15T00:35:13.130729+0000","last_active":"2024-09-15T01:11:49.137518+0000","last_peered":"2024-09-15T01:11:49.137518+0000","last_clean":"2024-09-15T01:11:49.137518+0000","last_became_active":"2024-09-15T00:35:11.119438+0000","last_became_peered":"2024-09-15T00:35:11.119438+0000","last_unstale":"2024-09-15T01:11:49.137518+0000","last_undegraded":"2024-09-15T01:11:49.137518+0000","last_fullsized":"2024-09-15T01:11:49.137518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:02:21.034948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071155299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1143,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.136645+0000","last_change":"2024-09-15T00:35:06.101340+0000","last_active":"2024-09-15T01:11:49.136645+0000","last_peered":"2024-09-15T01:11:49.136645+0000","last_clean":"2024-09-15T01:11:49.136645+0000","last_became_active":"2024-09-15T00:35:06.100362+0000","last_became_peered":"2024-09-15T00:35:06.100362+0000","last_unstale":"2024-09-15T01:11:49.136645+0000","last_undegraded":"2024-09-15T01:11:49.136645+0000","last_fullsized":"2024-09-15T01:11:49.136645+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_clean_scrub_stamp":"2024-09-15T00:35:05.089463+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:04.296940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706111,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1454268,"kb_used_data":5864,"kb_used_omap":1,"kb_used_meta":1448382,"kb_avail":92917572,"statfs":{"total":96636764160,"available":95147593728,"internally_reserved":0,"allocated":6004736,"data_stored":2994664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1483143621},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1476344,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":1470398,"kb_avail":92895496,"statfs":{"total":96636764160,"available":95124987904,"internally_reserved":0,"allocated":6066176,"data_stored":3585051,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1505688005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1603428,"kb_used_data":5908,"kb_used_omap":1,"kb_used_meta":1597502,"kb_avail":92768412,"statfs":{"total":96636764160,"available":94994853888,"internally_reserved":0,"allocated":6049792,"data_stored":3577909,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1635842501},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:11:53.089 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-15T01:11:53.377 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T01:11:53.377 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":507,"fsid":"201f1975-c83a-4da9-9090-536881f2f3b1","created":"2024-09-15T00:34:54.600851+0000","modified":"2024-09-15T01:11:50.988823+0000","last_up_change":"2024-09-15T00:35:04.086315+0000","last_in_change":"2024-09-15T00:34:55.456180+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T00:35:04.120349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T00:35:09.796195+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8631fc02-9239-4ed9-a93e-68265cd02da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6809","nonce":4037059923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6811","nonce":4037059923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6815","nonce":4037059923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":4037059923},{"type":"v1","addr":"172.21.15.59:6813","nonce":4037059923}]},"public_addr":"172.21.15.59:6809/4037059923","cluster_addr":"172.21.15.59:6811/4037059923","heartbeat_back_addr":"172.21.15.59:6815/4037059923","heartbeat_front_addr":"172.21.15.59:6813/4037059923","state":["exists","up"]},{"osd":1,"uuid":"1d919c9a-dc94-45ee-b684-fe85740c1c3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6801","nonce":710469187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6803","nonce":710469187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6807","nonce":710469187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":710469187},{"type":"v1","addr":"172.21.15.59:6805","nonce":710469187}]},"public_addr":"172.21.15.59:6801/710469187","cluster_addr":"172.21.15.59:6803/710469187","heartbeat_back_addr":"172.21.15.59:6807/710469187","heartbeat_front_addr":"172.21.15.59:6805/710469187","state":["exists","up"]},{"osd":2,"uuid":"3a30c237-ccd1-4b8e-9566-7e5f7c9f025f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6817","nonce":1104374936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6819","nonce":1104374936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6823","nonce":1104374936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1104374936},{"type":"v1","addr":"172.21.15.59:6821","nonce":1104374936}]},"public_addr":"172.21.15.59:6817/1104374936","cluster_addr":"172.21.15.59:6819/1104374936","heartbeat_back_addr":"172.21.15.59:6823/1104374936","heartbeat_front_addr":"172.21.15.59:6821/1104374936","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:01.699919+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:01.184721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:35:02.247268+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/51682475":"2024-09-15T02:11:41.955380+0000","172.21.15.59:0/293985851":"2024-09-15T02:10:59.708308+0000","172.21.15.59:0/771229832":"2024-09-15T02:10:16.226072+0000","172.21.15.59:0/2666518619":"2024-09-15T02:09:33.104535+0000","172.21.15.59:0/57230525":"2024-09-15T02:08:11.909298+0000","172.21.15.59:0/3122590992":"2024-09-15T02:06:03.007712+0000","172.21.15.59:0/2128157668":"2024-09-15T02:05:19.977097+0000","172.21.15.59:0/1798973270":"2024-09-15T02:01:57.648969+0000","172.21.15.59:0/1581398874":"2024-09-15T01:58:01.973672+0000","172.21.15.59:0/1135766921":"2024-09-15T02:00:35.074794+0000","172.21.15.59:0/1587749320":"2024-09-15T01:57:39.125882+0000","172.21.15.59:0/463390946":"2024-09-15T02:01:11.582443+0000","172.21.15.59:0/4048121280":"2024-09-15T01:57:32.117878+0000","172.21.15.59:0/360988029":"2024-09-15T01:55:47.157606+0000","172.21.15.59:0/2170918991":"2024-09-15T01:57:18.076841+0000","172.21.15.59:0/808590289":"2024-09-15T01:42:38.460913+0000","172.21.15.59:0/156481515":"2024-09-15T01:57:04.041242+0000","172.21.15.59:0/889282584":"2024-09-15T02:00:21.779146+0000","172.21.15.59:0/3779656291":"2024-09-15T01:56:41.492774+0000","172.21.15.59:0/3773508839":"2024-09-15T01:56:17.438093+0000","172.21.15.59:0/3881420872":"2024-09-15T02:08:55.267028+0000","172.21.15.59:0/3334164433":"2024-09-15T01:56:33.487209+0000","172.21.15.59:0/1961509071":"2024-09-15T01:57:25.283260+0000","172.21.15.59:0/2219957716":"2024-09-15T02:07:28.015739+0000","172.21.15.59:0/2588765547":"2024-09-15T02:00:49.106047+0000","172.21.15.59:0/3810440388":"2024-09-15T01:41:51.873644+0000","172.21.15.59:0/2977680089":"2024-09-15T01:56:09.441735+0000","172.21.15.59:0/2067140381":"2024-09-15T01:58:42.611653+0000","172.21.15.59:0/2139172736":"2024-09-15T01:37:26.680306+0000","172.21.15.59:0/995054729":"2024-09-15T01:57:11.046441+0000","172.21.15.59:0/4061884573":"2024-09-15T01:57:55.459315+0000","172.21.15.59:0/1178388234":"2024-09-15T01:57:47.453213+0000","172.21.15.59:0/45182101":"2024-09-15T01:38:20.865503+0000","172.21.15.59:0/2666984154":"2024-09-15T01:58:56.137209+0000","172.21.15.59:0/2044022711":"2024-09-15T01:59:56.494045+0000","172.21.15.59:0/4211951227":"2024-09-15T02:01:48.967130+0000","172.21.15.59:0/517583936":"2024-09-15T01:39:17.065029+0000","172.21.15.59:0/4200276692":"2024-09-15T01:58:26.560566+0000","172.21.15.59:0/1200224053":"2024-09-15T01:40:10.913731+0000","172.21.15.59:0/1538895504":"2024-09-15T02:01:34.609883+0000","172.21.15.59:0/571817594":"2024-09-15T01:59:03.053321+0000","172.21.15.59:0/738588831":"2024-09-15T01:43:31.336197+0000","172.21.15.59:0/2114580825":"2024-09-15T02:06:50.014227+0000","172.21.15.59:0/2132514601":"2024-09-15T01:41:03.649382+0000","172.21.15.59:0/61372321":"2024-09-15T01:45:23.704883+0000","172.21.15.59:0/4033816102":"2024-09-15T01:56:01.090420+0000","172.21.15.59:0/563537486":"2024-09-15T01:56:57.517956+0000","172.21.15.59:0/977336105":"2024-09-15T01:55:54.118086+0000","172.21.15.59:0/797729386":"2024-09-15T02:01:42.634181+0000","172.21.15.59:0/4102605718":"2024-09-15T01:58:10.512767+0000","172.21.15.59:0/46292483":"2024-09-15T01:58:18.544476+0000","172.21.15.59:0/2557621531":"2024-09-15T01:58:34.552613+0000","172.21.15.59:0/2351816154":"2024-09-15T02:01:26.094681+0000","172.21.15.59:0/1064938076":"2024-09-15T01:58:48.937751+0000","172.21.15.59:0/715856406":"2024-09-15T01:59:10.075806+0000","172.21.15.59:0/1514146209":"2024-09-15T01:59:18.728125+0000","172.21.15.59:0/2727996173":"2024-09-15T01:59:25.137962+0000","172.21.15.59:0/3439017484":"2024-09-15T02:00:04.491206+0000","172.21.15.59:0/2263715433":"2024-09-15T01:59:32.304514+0000","172.21.15.59:0/1638677675":"2024-09-15T01:59:40.268651+0000","172.21.15.59:0/3578208697":"2024-09-15T01:59:48.473954+0000","172.21.15.59:0/2725275114":"2024-09-15T02:00:42.129415+0000","172.21.15.59:0/608110914":"2024-09-15T02:01:03.161598+0000","172.21.15.59:0/3726109757":"2024-09-15T02:00:12.776855+0000","172.21.15.59:0/2448342131":"2024-09-15T02:00:28.023676+0000","172.21.15.59:0/2957901611":"2024-09-15T02:00:56.149716+0000","172.21.15.59:0/3993263867":"2024-09-15T01:56:25.505762+0000","172.21.15.59:0/1353869131":"2024-09-15T01:44:28.412104+0000","172.21.15.59:0/4015770178":"2024-09-15T01:56:49.514350+0000","172.21.15.59:0/3319578752":"2024-09-15T02:01:19.603790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":194,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T01:11:54.391 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-15T01:11:54.392 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T01:11:54.532 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-15T01:11:54.532 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T01:11:54.532 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-15T01:11:54.543 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-15T01:11:54.837 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T01:11:54.850 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-15T01:11:54.851 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T01:11:54.977 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-15T01:11:54.977 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T01:11:54.977 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-15T01:11:54.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-15T01:11:55.277 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T01:11:55.290 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-15T01:11:55.290 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T01:11:55.416 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-15T01:11:55.417 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T01:11:55.417 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-15T01:11:55.427 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-15T01:11:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T01:11:55.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-15T01:11:56.022 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T01:11:56.022 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-15T01:11:56.034 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":1610,"stamp":"2024-09-15T01:11:54.670915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4534040,"kb_used_data":17696,"kb_used_omap":4,"kb_used_meta":4516283,"kb_avail":278581480,"statfs":{"total":289910292480,"available":285267435520,"internally_reserved":0,"allocated":18120704,"data_stored":10157624,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4624674127},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-136474,"num_read_kb":-39664865,"num_write":-647873,"num_write_kb":-236761042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-24755248,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002215"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.138634+0000","last_change":"2024-09-15T00:35:13.131214+0000","last_active":"2024-09-15T01:11:49.138634+0000","last_peered":"2024-09-15T01:11:49.138634+0000","last_clean":"2024-09-15T01:11:49.138634+0000","last_became_active":"2024-09-15T00:35:11.119667+0000","last_became_peered":"2024-09-15T00:35:11.119667+0000","last_unstale":"2024-09-15T01:11:49.138634+0000","last_undegraded":"2024-09-15T01:11:49.138634+0000","last_fullsized":"2024-09-15T01:11:49.138634+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:56:15.752164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125682,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.145134+0000","last_change":"2024-09-15T00:35:13.131275+0000","last_active":"2024-09-15T01:11:49.145134+0000","last_peered":"2024-09-15T01:11:49.145134+0000","last_clean":"2024-09-15T01:11:49.145134+0000","last_became_active":"2024-09-15T00:35:11.119728+0000","last_became_peered":"2024-09-15T00:35:11.119728+0000","last_unstale":"2024-09-15T01:11:49.145134+0000","last_undegraded":"2024-09-15T01:11:49.145134+0000","last_fullsized":"2024-09-15T01:11:49.145134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:38:32.666290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193798,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.138361+0000","last_change":"2024-09-15T00:35:13.131573+0000","last_active":"2024-09-15T01:11:49.138361+0000","last_peered":"2024-09-15T01:11:49.138361+0000","last_clean":"2024-09-15T01:11:49.138361+0000","last_became_active":"2024-09-15T00:35:11.120732+0000","last_became_peered":"2024-09-15T00:35:11.120732+0000","last_unstale":"2024-09-15T01:11:49.138361+0000","last_undegraded":"2024-09-15T01:11:49.138361+0000","last_fullsized":"2024-09-15T01:11:49.138361+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:01:09.277084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001482562,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.142798+0000","last_change":"2024-09-15T00:35:13.130700+0000","last_active":"2024-09-15T01:11:49.142798+0000","last_peered":"2024-09-15T01:11:49.142798+0000","last_clean":"2024-09-15T01:11:49.142798+0000","last_became_active":"2024-09-15T00:35:11.120070+0000","last_became_peered":"2024-09-15T00:35:11.120070+0000","last_unstale":"2024-09-15T01:11:49.142798+0000","last_undegraded":"2024-09-15T01:11:49.142798+0000","last_fullsized":"2024-09-15T01:11:49.142798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:26:49.101024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070255600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1016,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-15T01:11:42.989247+0000","last_change":"2024-09-15T00:35:13.128765+0000","last_active":"2024-09-15T01:11:42.989247+0000","last_peered":"2024-09-15T01:11:42.989247+0000","last_clean":"2024-09-15T01:11:42.989247+0000","last_became_active":"2024-09-15T00:35:11.120934+0000","last_became_peered":"2024-09-15T00:35:11.120934+0000","last_unstale":"2024-09-15T01:11:42.989247+0000","last_undegraded":"2024-09-15T01:11:42.989247+0000","last_fullsized":"2024-09-15T01:11:42.989247+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:45:31.683827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011530049999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.006342+0000","last_change":"2024-09-15T00:35:13.345445+0000","last_active":"2024-09-15T01:11:49.006342+0000","last_peered":"2024-09-15T01:11:49.006342+0000","last_clean":"2024-09-15T01:11:49.006342+0000","last_became_active":"2024-09-15T00:35:11.123441+0000","last_became_peered":"2024-09-15T00:35:11.123441+0000","last_unstale":"2024-09-15T01:11:49.006342+0000","last_undegraded":"2024-09-15T01:11:49.006342+0000","last_fullsized":"2024-09-15T01:11:49.006342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:34:52.254834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000350637,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.005803+0000","last_change":"2024-09-15T00:35:13.345476+0000","last_active":"2024-09-15T01:11:49.005803+0000","last_peered":"2024-09-15T01:11:49.005803+0000","last_clean":"2024-09-15T01:11:49.005803+0000","last_became_active":"2024-09-15T00:35:11.123456+0000","last_became_peered":"2024-09-15T00:35:11.123456+0000","last_unstale":"2024-09-15T01:11:49.005803+0000","last_undegraded":"2024-09-15T01:11:49.005803+0000","last_fullsized":"2024-09-15T01:11:49.005803+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:43.436651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038329899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.137518+0000","last_change":"2024-09-15T00:35:13.130729+0000","last_active":"2024-09-15T01:11:49.137518+0000","last_peered":"2024-09-15T01:11:49.137518+0000","last_clean":"2024-09-15T01:11:49.137518+0000","last_became_active":"2024-09-15T00:35:11.119438+0000","last_became_peered":"2024-09-15T00:35:11.119438+0000","last_unstale":"2024-09-15T01:11:49.137518+0000","last_undegraded":"2024-09-15T01:11:49.137518+0000","last_fullsized":"2024-09-15T01:11:49.137518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:10.101939+0000","last_clean_scrub_stamp":"2024-09-15T00:35:10.101939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:02:21.034948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071155299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1143,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-15T01:11:49.136645+0000","last_change":"2024-09-15T00:35:06.101340+0000","last_active":"2024-09-15T01:11:49.136645+0000","last_peered":"2024-09-15T01:11:49.136645+0000","last_clean":"2024-09-15T01:11:49.136645+0000","last_became_active":"2024-09-15T00:35:06.100362+0000","last_became_peered":"2024-09-15T00:35:06.100362+0000","last_unstale":"2024-09-15T01:11:49.136645+0000","last_undegraded":"2024-09-15T01:11:49.136645+0000","last_fullsized":"2024-09-15T01:11:49.136645+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:35:05.089463+0000","last_clean_scrub_stamp":"2024-09-15T00:35:05.089463+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:02:04.296940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706111,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1454268,"kb_used_data":5864,"kb_used_omap":1,"kb_used_meta":1448382,"kb_avail":92917572,"statfs":{"total":96636764160,"available":95147593728,"internally_reserved":0,"allocated":6004736,"data_stored":2994664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1483143621},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1476344,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":1470398,"kb_avail":92895496,"statfs":{"total":96636764160,"available":95124987904,"internally_reserved":0,"allocated":6066176,"data_stored":3585051,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1505688005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1603428,"kb_used_data":5908,"kb_used_omap":1,"kb_used_meta":1597502,"kb_avail":92768412,"statfs":{"total":96636764160,"available":94994853888,"internally_reserved":0,"allocated":6049792,"data_stored":3577909,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1635842501},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:11:56.038 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-15T00:35:10.101939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=35, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=11, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:11:56.038 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-15T00:35:10.101939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=35, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=11, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:11:56.039 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-15T00:35:10.101939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=35, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=11, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:11:56.039 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-15T00:35:10.101939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=35, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=11, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:11:56.039 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-15T00:35:10.101939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=35, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=11, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:11:56.039 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-15T00:35:10.101939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=35, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=11, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:11:56.039 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T00:35:10.101939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=35, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=11, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:11:56.039 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-15T00:35:10.101939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=35, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=11, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:11:56.039 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T00:35:05.089463+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=35, tm_sec=5, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=11, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:11:56.040 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T01:12:16.041 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-15T01:12:16.342 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-15T01:12:16.342 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-15T01:12:16.354 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2024-09-15T01:12:14.674794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4517444,"kb_used_data":17748,"kb_used_omap":4,"kb_used_meta":4499643,"kb_avail":278598076,"statfs":{"total":289910292480,"available":285284429824,"internally_reserved":0,"allocated":18173952,"data_stored":10184713,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4607634767},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002503"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1025,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-15T01:11:58.949198+0000","last_change":"2024-09-15T01:11:58.949198+0000","last_active":"2024-09-15T01:11:58.949198+0000","last_peered":"2024-09-15T01:11:58.949198+0000","last_clean":"2024-09-15T01:11:58.949198+0000","last_became_active":"2024-09-15T00:35:11.119667+0000","last_became_peered":"2024-09-15T00:35:11.119667+0000","last_unstale":"2024-09-15T01:11:58.949198+0000","last_undegraded":"2024-09-15T01:11:58.949198+0000","last_fullsized":"2024-09-15T01:11:58.949198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:11:58.949076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:11:58.949076+0000","last_clean_scrub_stamp":"2024-09-15T01:11:58.949076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:43:12.500162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125682,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1025,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-15T01:11:57.979461+0000","last_change":"2024-09-15T01:11:57.979461+0000","last_active":"2024-09-15T01:11:57.979461+0000","last_peered":"2024-09-15T01:11:57.979461+0000","last_clean":"2024-09-15T01:11:57.979461+0000","last_became_active":"2024-09-15T00:35:11.119728+0000","last_became_peered":"2024-09-15T00:35:11.119728+0000","last_unstale":"2024-09-15T01:11:57.979461+0000","last_undegraded":"2024-09-15T01:11:57.979461+0000","last_fullsized":"2024-09-15T01:11:57.979461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:11:57.979286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:11:57.979286+0000","last_clean_scrub_stamp":"2024-09-15T01:11:57.979286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:09:38.179262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193798,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1025,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-15T01:12:01.035535+0000","last_change":"2024-09-15T01:12:01.035561+0000","last_active":"2024-09-15T01:12:01.035535+0000","last_peered":"2024-09-15T01:12:01.035535+0000","last_clean":"2024-09-15T01:12:01.035535+0000","last_became_active":"2024-09-15T00:35:11.120732+0000","last_became_peered":"2024-09-15T00:35:11.120732+0000","last_unstale":"2024-09-15T01:12:01.035535+0000","last_undegraded":"2024-09-15T01:12:01.035535+0000","last_fullsized":"2024-09-15T01:12:01.035535+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:12:01.035438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:12:01.035438+0000","last_clean_scrub_stamp":"2024-09-15T01:12:01.035438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:41:09.127222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001482562,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1025,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-15T01:11:59.985963+0000","last_change":"2024-09-15T01:11:59.985992+0000","last_active":"2024-09-15T01:11:59.985963+0000","last_peered":"2024-09-15T01:11:59.985963+0000","last_clean":"2024-09-15T01:11:59.985963+0000","last_became_active":"2024-09-15T00:35:11.120070+0000","last_became_peered":"2024-09-15T00:35:11.120070+0000","last_unstale":"2024-09-15T01:11:59.985963+0000","last_undegraded":"2024-09-15T01:11:59.985963+0000","last_fullsized":"2024-09-15T01:11:59.985963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:11:59.985854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:11:59.985854+0000","last_clean_scrub_stamp":"2024-09-15T01:11:59.985854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:54:24.289755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070255600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"507'3","reported_seq":1030,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-15T01:11:55.563040+0000","last_change":"2024-09-15T01:11:55.563083+0000","last_active":"2024-09-15T01:11:55.563040+0000","last_peered":"2024-09-15T01:11:55.563040+0000","last_clean":"2024-09-15T01:11:55.563040+0000","last_became_active":"2024-09-15T00:35:11.120934+0000","last_became_peered":"2024-09-15T00:35:11.120934+0000","last_unstale":"2024-09-15T01:11:55.563040+0000","last_undegraded":"2024-09-15T01:11:55.563040+0000","last_fullsized":"2024-09-15T01:11:55.563040+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'3","last_scrub_stamp":"2024-09-15T01:11:55.562924+0000","last_deep_scrub":"507'3","last_deep_scrub_stamp":"2024-09-15T01:11:55.562924+0000","last_clean_scrub_stamp":"2024-09-15T01:11:55.562924+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:56:09.463518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011530049999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1027,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-15T01:11:56.103733+0000","last_change":"2024-09-15T01:11:56.103751+0000","last_active":"2024-09-15T01:11:56.103733+0000","last_peered":"2024-09-15T01:11:56.103733+0000","last_clean":"2024-09-15T01:11:56.103733+0000","last_became_active":"2024-09-15T00:35:11.123441+0000","last_became_peered":"2024-09-15T00:35:11.123441+0000","last_unstale":"2024-09-15T01:11:56.103733+0000","last_undegraded":"2024-09-15T01:11:56.103733+0000","last_fullsized":"2024-09-15T01:11:56.103733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:11:56.103652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:11:56.103652+0000","last_clean_scrub_stamp":"2024-09-15T01:11:56.103652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:48:53.819522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000350637,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1028,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-15T01:11:57.076375+0000","last_change":"2024-09-15T01:11:57.076401+0000","last_active":"2024-09-15T01:11:57.076375+0000","last_peered":"2024-09-15T01:11:57.076375+0000","last_clean":"2024-09-15T01:11:57.076375+0000","last_became_active":"2024-09-15T00:35:11.123456+0000","last_became_peered":"2024-09-15T00:35:11.123456+0000","last_unstale":"2024-09-15T01:11:57.076375+0000","last_undegraded":"2024-09-15T01:11:57.076375+0000","last_fullsized":"2024-09-15T01:11:57.076375+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:11:57.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:11:57.076275+0000","last_clean_scrub_stamp":"2024-09-15T01:11:57.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:14:31.684211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038329899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"507'2","reported_seq":1027,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-15T01:11:56.942612+0000","last_change":"2024-09-15T01:11:56.942640+0000","last_active":"2024-09-15T01:11:56.942612+0000","last_peered":"2024-09-15T01:11:56.942612+0000","last_clean":"2024-09-15T01:11:56.942612+0000","last_became_active":"2024-09-15T00:35:11.119438+0000","last_became_peered":"2024-09-15T00:35:11.119438+0000","last_unstale":"2024-09-15T01:11:56.942612+0000","last_undegraded":"2024-09-15T01:11:56.942612+0000","last_fullsized":"2024-09-15T01:11:56.942612+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'2","last_scrub_stamp":"2024-09-15T01:11:56.942519+0000","last_deep_scrub":"507'2","last_deep_scrub_stamp":"2024-09-15T01:11:56.942519+0000","last_clean_scrub_stamp":"2024-09-15T01:11:56.942519+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:01:07.623594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071155299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"507'99","reported_seq":1156,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-15T01:11:55.929149+0000","last_change":"2024-09-15T01:11:55.929224+0000","last_active":"2024-09-15T01:11:55.929149+0000","last_peered":"2024-09-15T01:11:55.929149+0000","last_clean":"2024-09-15T01:11:55.929149+0000","last_became_active":"2024-09-15T00:35:06.100362+0000","last_became_peered":"2024-09-15T00:35:06.100362+0000","last_unstale":"2024-09-15T01:11:55.929149+0000","last_undegraded":"2024-09-15T01:11:55.929149+0000","last_fullsized":"2024-09-15T01:11:55.929149+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'99","last_scrub_stamp":"2024-09-15T01:11:55.929026+0000","last_deep_scrub":"507'99","last_deep_scrub_stamp":"2024-09-15T01:11:55.929026+0000","last_clean_scrub_stamp":"2024-09-15T01:11:55.929026+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:37:48.996250+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11626,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706115,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1454276,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1448382,"kb_avail":92917564,"statfs":{"total":96636764160,"available":95147585536,"internally_reserved":0,"allocated":6017024,"data_stored":3001313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1483143621},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1476352,"kb_used_data":5936,"kb_used_omap":1,"kb_used_meta":1470398,"kb_avail":92895488,"statfs":{"total":96636764160,"available":95124979712,"internally_reserved":0,"allocated":6078464,"data_stored":3591700,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1505688005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738818,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1586816,"kb_used_data":5936,"kb_used_omap":1,"kb_used_meta":1580862,"kb_avail":92785024,"statfs":{"total":96636764160,"available":95011864576,"internally_reserved":0,"allocated":6078464,"data_stored":3591700,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1618803141},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5813,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:12:16.355 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-15T01:12:16.662 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T01:12:16.662 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T01:12:16.662 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-15T01:12:16.663 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:12:16.731 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-15T01:12:16.731 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-15T01:12:16.731 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:12:16.801 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-15T01:12:16.802 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-15T01:12:16.802 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:12:16.874 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-15T01:12:16.874 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T01:12:16.875 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-15T01:12:16.875 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:12:16.905 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-15T01:12:16.905 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T01:12:16.905 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-15T01:12:16.906 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:12:16.918 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-15T01:12:16.918 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T01:12:16.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-15T01:12:17.019 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T01:12:17.020 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-15T01:12:17.325 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T01:12:17.325 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-15T01:12:17.461 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T01:12:17.461 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-15T01:12:17.593 INFO:tasks.ceph:Archiving mon data... 2024-09-15T01:12:17.594 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845/data/cluster2.mon.a.tgz 2024-09-15T01:12:17.594 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-09-15T01:12:17.612 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.Yy4PDmjuZu 2024-09-15T01:12:17.612 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.Yy4PDmjuZu 2024-09-15T01:12:17.795 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.Yy4PDmjuZu 2024-09-15T01:12:17.908 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.Yy4PDmjuZu is 479KB 2024-09-15T01:12:18.007 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.Yy4PDmjuZu 2024-09-15T01:12:18.022 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T01:12:18.023 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T01:12:18.026 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T01:12:18.088 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T01:12:18.107 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-15T01:12:18.108 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-15T01:12:18.408 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T01:12:18.409 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-15T01:12:18.421 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":1601,"stamp":"2024-09-15T01:12:16.902475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":460204,"kb_used_data":11376,"kb_used_omap":4,"kb_used_meta":448763,"kb_avail":282655316,"statfs":{"total":289910292480,"available":289439043584,"internally_reserved":0,"allocated":11649024,"data_stored":7349014,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":459533647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.189758+0000","last_change":"2024-09-15T00:34:34.578484+0000","last_active":"2024-09-15T01:11:50.189758+0000","last_peered":"2024-09-15T01:11:50.189758+0000","last_clean":"2024-09-15T01:11:50.189758+0000","last_became_active":"2024-09-15T00:34:32.575307+0000","last_became_peered":"2024-09-15T00:34:32.575307+0000","last_unstale":"2024-09-15T01:11:50.189758+0000","last_undegraded":"2024-09-15T01:11:50.189758+0000","last_fullsized":"2024-09-15T01:11:50.189758+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:04:44.235727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010984479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":940,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.192191+0000","last_change":"2024-09-15T00:34:34.578159+0000","last_active":"2024-09-15T01:11:50.192191+0000","last_peered":"2024-09-15T01:11:50.192191+0000","last_clean":"2024-09-15T01:11:50.192191+0000","last_became_active":"2024-09-15T00:34:32.576552+0000","last_became_peered":"2024-09-15T00:34:32.576552+0000","last_unstale":"2024-09-15T01:11:50.192191+0000","last_undegraded":"2024-09-15T01:11:50.192191+0000","last_fullsized":"2024-09-15T01:11:50.192191+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:22.053783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001076133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191453+0000","last_change":"2024-09-15T00:34:34.577770+0000","last_active":"2024-09-15T01:11:50.191453+0000","last_peered":"2024-09-15T01:11:50.191453+0000","last_clean":"2024-09-15T01:11:50.191453+0000","last_became_active":"2024-09-15T00:34:32.574697+0000","last_became_peered":"2024-09-15T00:34:32.574697+0000","last_unstale":"2024-09-15T01:11:50.191453+0000","last_undegraded":"2024-09-15T01:11:50.191453+0000","last_fullsized":"2024-09-15T01:11:50.191453+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:56:34.709213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073585800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191649+0000","last_change":"2024-09-15T00:34:34.578167+0000","last_active":"2024-09-15T01:11:50.191649+0000","last_peered":"2024-09-15T01:11:50.191649+0000","last_clean":"2024-09-15T01:11:50.191649+0000","last_became_active":"2024-09-15T00:34:32.575384+0000","last_became_peered":"2024-09-15T00:34:32.575384+0000","last_unstale":"2024-09-15T01:11:50.191649+0000","last_undegraded":"2024-09-15T01:11:50.191649+0000","last_fullsized":"2024-09-15T01:11:50.191649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:06:32.098451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010254979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":943,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.660903+0000","last_change":"2024-09-15T00:34:34.574756+0000","last_active":"2024-09-15T01:11:52.660903+0000","last_peered":"2024-09-15T01:11:52.660903+0000","last_clean":"2024-09-15T01:11:52.660903+0000","last_became_active":"2024-09-15T00:34:32.575639+0000","last_became_peered":"2024-09-15T00:34:32.575639+0000","last_unstale":"2024-09-15T01:11:52.660903+0000","last_undegraded":"2024-09-15T01:11:52.660903+0000","last_fullsized":"2024-09-15T01:11:52.660903+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:44:05.451379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054626299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.661130+0000","last_change":"2024-09-15T00:34:34.912915+0000","last_active":"2024-09-15T01:11:52.661130+0000","last_peered":"2024-09-15T01:11:52.661130+0000","last_clean":"2024-09-15T01:11:52.661130+0000","last_became_active":"2024-09-15T00:34:32.573975+0000","last_became_peered":"2024-09-15T00:34:32.573975+0000","last_unstale":"2024-09-15T01:11:52.661130+0000","last_undegraded":"2024-09-15T01:11:52.661130+0000","last_fullsized":"2024-09-15T01:11:52.661130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:42.456821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040528,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.662529+0000","last_change":"2024-09-15T00:34:34.912807+0000","last_active":"2024-09-15T01:11:52.662529+0000","last_peered":"2024-09-15T01:11:52.662529+0000","last_clean":"2024-09-15T01:11:52.662529+0000","last_became_active":"2024-09-15T00:34:32.573804+0000","last_became_peered":"2024-09-15T00:34:32.573804+0000","last_unstale":"2024-09-15T01:11:52.662529+0000","last_undegraded":"2024-09-15T01:11:52.662529+0000","last_fullsized":"2024-09-15T01:11:52.662529+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:29:40.374387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000336428,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":942,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191278+0000","last_change":"2024-09-15T00:34:34.577734+0000","last_active":"2024-09-15T01:11:50.191278+0000","last_peered":"2024-09-15T01:11:50.191278+0000","last_clean":"2024-09-15T01:11:50.191278+0000","last_became_active":"2024-09-15T00:34:32.573483+0000","last_became_peered":"2024-09-15T00:34:32.573483+0000","last_unstale":"2024-09-15T01:11:50.191278+0000","last_undegraded":"2024-09-15T01:11:50.191278+0000","last_fullsized":"2024-09-15T01:11:50.191278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:56:27.571672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067056699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1069,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.192800+0000","last_change":"2024-09-15T00:34:25.561004+0000","last_active":"2024-09-15T01:11:50.192800+0000","last_peered":"2024-09-15T01:11:50.192800+0000","last_clean":"2024-09-15T01:11:50.192800+0000","last_became_active":"2024-09-15T00:34:25.559814+0000","last_became_peered":"2024-09-15T00:34:25.559814+0000","last_unstale":"2024-09-15T01:11:50.192800+0000","last_undegraded":"2024-09-15T01:11:50.192800+0000","last_fullsized":"2024-09-15T01:11:50.192800+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_clean_scrub_stamp":"2024-09-15T00:34:24.549251+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:04:58.147756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706124,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165432,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206408,"statfs":{"total":96636764160,"available":96467361792,"internally_reserved":0,"allocated":3903488,"data_stored":2646467,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129276,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94242564,"statfs":{"total":96636764160,"available":96504385536,"internally_reserved":0,"allocated":3842048,"data_stored":2056080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738826,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165496,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206344,"statfs":{"total":96636764160,"available":96467296256,"internally_reserved":0,"allocated":3903488,"data_stored":2646467,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:12:18.422 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-15T01:12:18.707 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T01:12:18.708 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-15T01:12:18.720 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":1601,"stamp":"2024-09-15T01:12:16.902475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":460204,"kb_used_data":11376,"kb_used_omap":4,"kb_used_meta":448763,"kb_avail":282655316,"statfs":{"total":289910292480,"available":289439043584,"internally_reserved":0,"allocated":11649024,"data_stored":7349014,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":459533647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.189758+0000","last_change":"2024-09-15T00:34:34.578484+0000","last_active":"2024-09-15T01:11:50.189758+0000","last_peered":"2024-09-15T01:11:50.189758+0000","last_clean":"2024-09-15T01:11:50.189758+0000","last_became_active":"2024-09-15T00:34:32.575307+0000","last_became_peered":"2024-09-15T00:34:32.575307+0000","last_unstale":"2024-09-15T01:11:50.189758+0000","last_undegraded":"2024-09-15T01:11:50.189758+0000","last_fullsized":"2024-09-15T01:11:50.189758+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:04:44.235727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010984479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":940,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.192191+0000","last_change":"2024-09-15T00:34:34.578159+0000","last_active":"2024-09-15T01:11:50.192191+0000","last_peered":"2024-09-15T01:11:50.192191+0000","last_clean":"2024-09-15T01:11:50.192191+0000","last_became_active":"2024-09-15T00:34:32.576552+0000","last_became_peered":"2024-09-15T00:34:32.576552+0000","last_unstale":"2024-09-15T01:11:50.192191+0000","last_undegraded":"2024-09-15T01:11:50.192191+0000","last_fullsized":"2024-09-15T01:11:50.192191+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:22.053783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001076133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191453+0000","last_change":"2024-09-15T00:34:34.577770+0000","last_active":"2024-09-15T01:11:50.191453+0000","last_peered":"2024-09-15T01:11:50.191453+0000","last_clean":"2024-09-15T01:11:50.191453+0000","last_became_active":"2024-09-15T00:34:32.574697+0000","last_became_peered":"2024-09-15T00:34:32.574697+0000","last_unstale":"2024-09-15T01:11:50.191453+0000","last_undegraded":"2024-09-15T01:11:50.191453+0000","last_fullsized":"2024-09-15T01:11:50.191453+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:56:34.709213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073585800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191649+0000","last_change":"2024-09-15T00:34:34.578167+0000","last_active":"2024-09-15T01:11:50.191649+0000","last_peered":"2024-09-15T01:11:50.191649+0000","last_clean":"2024-09-15T01:11:50.191649+0000","last_became_active":"2024-09-15T00:34:32.575384+0000","last_became_peered":"2024-09-15T00:34:32.575384+0000","last_unstale":"2024-09-15T01:11:50.191649+0000","last_undegraded":"2024-09-15T01:11:50.191649+0000","last_fullsized":"2024-09-15T01:11:50.191649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:06:32.098451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010254979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":943,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.660903+0000","last_change":"2024-09-15T00:34:34.574756+0000","last_active":"2024-09-15T01:11:52.660903+0000","last_peered":"2024-09-15T01:11:52.660903+0000","last_clean":"2024-09-15T01:11:52.660903+0000","last_became_active":"2024-09-15T00:34:32.575639+0000","last_became_peered":"2024-09-15T00:34:32.575639+0000","last_unstale":"2024-09-15T01:11:52.660903+0000","last_undegraded":"2024-09-15T01:11:52.660903+0000","last_fullsized":"2024-09-15T01:11:52.660903+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:44:05.451379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054626299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.661130+0000","last_change":"2024-09-15T00:34:34.912915+0000","last_active":"2024-09-15T01:11:52.661130+0000","last_peered":"2024-09-15T01:11:52.661130+0000","last_clean":"2024-09-15T01:11:52.661130+0000","last_became_active":"2024-09-15T00:34:32.573975+0000","last_became_peered":"2024-09-15T00:34:32.573975+0000","last_unstale":"2024-09-15T01:11:52.661130+0000","last_undegraded":"2024-09-15T01:11:52.661130+0000","last_fullsized":"2024-09-15T01:11:52.661130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:42.456821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040528,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.662529+0000","last_change":"2024-09-15T00:34:34.912807+0000","last_active":"2024-09-15T01:11:52.662529+0000","last_peered":"2024-09-15T01:11:52.662529+0000","last_clean":"2024-09-15T01:11:52.662529+0000","last_became_active":"2024-09-15T00:34:32.573804+0000","last_became_peered":"2024-09-15T00:34:32.573804+0000","last_unstale":"2024-09-15T01:11:52.662529+0000","last_undegraded":"2024-09-15T01:11:52.662529+0000","last_fullsized":"2024-09-15T01:11:52.662529+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:29:40.374387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000336428,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":942,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191278+0000","last_change":"2024-09-15T00:34:34.577734+0000","last_active":"2024-09-15T01:11:50.191278+0000","last_peered":"2024-09-15T01:11:50.191278+0000","last_clean":"2024-09-15T01:11:50.191278+0000","last_became_active":"2024-09-15T00:34:32.573483+0000","last_became_peered":"2024-09-15T00:34:32.573483+0000","last_unstale":"2024-09-15T01:11:50.191278+0000","last_undegraded":"2024-09-15T01:11:50.191278+0000","last_fullsized":"2024-09-15T01:11:50.191278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:56:27.571672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067056699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1069,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.192800+0000","last_change":"2024-09-15T00:34:25.561004+0000","last_active":"2024-09-15T01:11:50.192800+0000","last_peered":"2024-09-15T01:11:50.192800+0000","last_clean":"2024-09-15T01:11:50.192800+0000","last_became_active":"2024-09-15T00:34:25.559814+0000","last_became_peered":"2024-09-15T00:34:25.559814+0000","last_unstale":"2024-09-15T01:11:50.192800+0000","last_undegraded":"2024-09-15T01:11:50.192800+0000","last_fullsized":"2024-09-15T01:11:50.192800+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_clean_scrub_stamp":"2024-09-15T00:34:24.549251+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:04:58.147756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706124,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165432,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206408,"statfs":{"total":96636764160,"available":96467361792,"internally_reserved":0,"allocated":3903488,"data_stored":2646467,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129276,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94242564,"statfs":{"total":96636764160,"available":96504385536,"internally_reserved":0,"allocated":3842048,"data_stored":2056080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738826,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165496,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206344,"statfs":{"total":96636764160,"available":96467296256,"internally_reserved":0,"allocated":3903488,"data_stored":2646467,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:12:18.720 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-15T01:12:18.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-15T01:12:19.008 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T01:12:19.009 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-15T01:12:19.021 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":1602,"stamp":"2024-09-15T01:12:18.902780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":460204,"kb_used_data":11376,"kb_used_omap":4,"kb_used_meta":448763,"kb_avail":282655316,"statfs":{"total":289910292480,"available":289439043584,"internally_reserved":0,"allocated":11649024,"data_stored":7349014,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":459533647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002462"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.189758+0000","last_change":"2024-09-15T00:34:34.578484+0000","last_active":"2024-09-15T01:11:50.189758+0000","last_peered":"2024-09-15T01:11:50.189758+0000","last_clean":"2024-09-15T01:11:50.189758+0000","last_became_active":"2024-09-15T00:34:32.575307+0000","last_became_peered":"2024-09-15T00:34:32.575307+0000","last_unstale":"2024-09-15T01:11:50.189758+0000","last_undegraded":"2024-09-15T01:11:50.189758+0000","last_fullsized":"2024-09-15T01:11:50.189758+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:04:44.235727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010984479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":940,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.192191+0000","last_change":"2024-09-15T00:34:34.578159+0000","last_active":"2024-09-15T01:11:50.192191+0000","last_peered":"2024-09-15T01:11:50.192191+0000","last_clean":"2024-09-15T01:11:50.192191+0000","last_became_active":"2024-09-15T00:34:32.576552+0000","last_became_peered":"2024-09-15T00:34:32.576552+0000","last_unstale":"2024-09-15T01:11:50.192191+0000","last_undegraded":"2024-09-15T01:11:50.192191+0000","last_fullsized":"2024-09-15T01:11:50.192191+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:22.053783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001076133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191453+0000","last_change":"2024-09-15T00:34:34.577770+0000","last_active":"2024-09-15T01:11:50.191453+0000","last_peered":"2024-09-15T01:11:50.191453+0000","last_clean":"2024-09-15T01:11:50.191453+0000","last_became_active":"2024-09-15T00:34:32.574697+0000","last_became_peered":"2024-09-15T00:34:32.574697+0000","last_unstale":"2024-09-15T01:11:50.191453+0000","last_undegraded":"2024-09-15T01:11:50.191453+0000","last_fullsized":"2024-09-15T01:11:50.191453+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:56:34.709213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073585800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191649+0000","last_change":"2024-09-15T00:34:34.578167+0000","last_active":"2024-09-15T01:11:50.191649+0000","last_peered":"2024-09-15T01:11:50.191649+0000","last_clean":"2024-09-15T01:11:50.191649+0000","last_became_active":"2024-09-15T00:34:32.575384+0000","last_became_peered":"2024-09-15T00:34:32.575384+0000","last_unstale":"2024-09-15T01:11:50.191649+0000","last_undegraded":"2024-09-15T01:11:50.191649+0000","last_fullsized":"2024-09-15T01:11:50.191649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:06:32.098451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010254979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":943,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.660903+0000","last_change":"2024-09-15T00:34:34.574756+0000","last_active":"2024-09-15T01:11:52.660903+0000","last_peered":"2024-09-15T01:11:52.660903+0000","last_clean":"2024-09-15T01:11:52.660903+0000","last_became_active":"2024-09-15T00:34:32.575639+0000","last_became_peered":"2024-09-15T00:34:32.575639+0000","last_unstale":"2024-09-15T01:11:52.660903+0000","last_undegraded":"2024-09-15T01:11:52.660903+0000","last_fullsized":"2024-09-15T01:11:52.660903+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:44:05.451379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054626299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.661130+0000","last_change":"2024-09-15T00:34:34.912915+0000","last_active":"2024-09-15T01:11:52.661130+0000","last_peered":"2024-09-15T01:11:52.661130+0000","last_clean":"2024-09-15T01:11:52.661130+0000","last_became_active":"2024-09-15T00:34:32.573975+0000","last_became_peered":"2024-09-15T00:34:32.573975+0000","last_unstale":"2024-09-15T01:11:52.661130+0000","last_undegraded":"2024-09-15T01:11:52.661130+0000","last_fullsized":"2024-09-15T01:11:52.661130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:42.456821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040528,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.662529+0000","last_change":"2024-09-15T00:34:34.912807+0000","last_active":"2024-09-15T01:11:52.662529+0000","last_peered":"2024-09-15T01:11:52.662529+0000","last_clean":"2024-09-15T01:11:52.662529+0000","last_became_active":"2024-09-15T00:34:32.573804+0000","last_became_peered":"2024-09-15T00:34:32.573804+0000","last_unstale":"2024-09-15T01:11:52.662529+0000","last_undegraded":"2024-09-15T01:11:52.662529+0000","last_fullsized":"2024-09-15T01:11:52.662529+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:29:40.374387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000336428,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":942,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191278+0000","last_change":"2024-09-15T00:34:34.577734+0000","last_active":"2024-09-15T01:11:50.191278+0000","last_peered":"2024-09-15T01:11:50.191278+0000","last_clean":"2024-09-15T01:11:50.191278+0000","last_became_active":"2024-09-15T00:34:32.573483+0000","last_became_peered":"2024-09-15T00:34:32.573483+0000","last_unstale":"2024-09-15T01:11:50.191278+0000","last_undegraded":"2024-09-15T01:11:50.191278+0000","last_fullsized":"2024-09-15T01:11:50.191278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:56:27.571672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067056699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1069,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.192800+0000","last_change":"2024-09-15T00:34:25.561004+0000","last_active":"2024-09-15T01:11:50.192800+0000","last_peered":"2024-09-15T01:11:50.192800+0000","last_clean":"2024-09-15T01:11:50.192800+0000","last_became_active":"2024-09-15T00:34:25.559814+0000","last_became_peered":"2024-09-15T00:34:25.559814+0000","last_unstale":"2024-09-15T01:11:50.192800+0000","last_undegraded":"2024-09-15T01:11:50.192800+0000","last_fullsized":"2024-09-15T01:11:50.192800+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_clean_scrub_stamp":"2024-09-15T00:34:24.549251+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:04:58.147756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706124,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165432,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206408,"statfs":{"total":96636764160,"available":96467361792,"internally_reserved":0,"allocated":3903488,"data_stored":2646467,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129276,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94242564,"statfs":{"total":96636764160,"available":96504385536,"internally_reserved":0,"allocated":3842048,"data_stored":2056080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738827,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165496,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206344,"statfs":{"total":96636764160,"available":96467296256,"internally_reserved":0,"allocated":3903488,"data_stored":2646467,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:12:19.022 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-15T01:12:19.312 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T01:12:19.312 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":467,"fsid":"189ebd14-4d6c-4cd0-845c-b433081070db","created":"2024-09-15T00:34:15.869593+0000","modified":"2024-09-15T01:11:50.183576+0000","last_up_change":"2024-09-15T00:34:24.549251+0000","last_in_change":"2024-09-15T00:34:16.726844+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T00:34:24.548876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T00:34:31.272465+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55e202c1-061d-4387-8c25-494b3be11d96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6801","nonce":3978604502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6803","nonce":3978604502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6807","nonce":3978604502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3978604502},{"type":"v1","addr":"172.21.15.2:6805","nonce":3978604502}]},"public_addr":"172.21.15.2:6801/3978604502","cluster_addr":"172.21.15.2:6803/3978604502","heartbeat_back_addr":"172.21.15.2:6807/3978604502","heartbeat_front_addr":"172.21.15.2:6805/3978604502","state":["exists","up"]},{"osd":1,"uuid":"3f34fa1a-c511-44ce-8b21-08ca50cc3b2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6817","nonce":3851036120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6819","nonce":3851036120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6823","nonce":3851036120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3851036120},{"type":"v1","addr":"172.21.15.2:6821","nonce":3851036120}]},"public_addr":"172.21.15.2:6817/3851036120","cluster_addr":"172.21.15.2:6819/3851036120","heartbeat_back_addr":"172.21.15.2:6823/3851036120","heartbeat_front_addr":"172.21.15.2:6821/3851036120","state":["exists","up"]},{"osd":2,"uuid":"7f402869-0489-40b8-8fa2-dea899922e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6809","nonce":3675309926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6811","nonce":3675309926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6815","nonce":3675309926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3675309926},{"type":"v1","addr":"172.21.15.2:6813","nonce":3675309926}]},"public_addr":"172.21.15.2:6809/3675309926","cluster_addr":"172.21.15.2:6811/3675309926","heartbeat_back_addr":"172.21.15.2:6815/3675309926","heartbeat_front_addr":"172.21.15.2:6813/3675309926","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:22.477886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:23.048083+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T00:34:22.268062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T01:12:20.327 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-15T01:12:20.327 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T01:12:20.468 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-15T01:12:20.468 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T01:12:20.468 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-15T01:12:20.479 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-15T01:12:20.768 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T01:12:20.782 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-15T01:12:20.782 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T01:12:20.909 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-15T01:12:20.909 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T01:12:20.909 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-15T01:12:20.920 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-15T01:12:21.208 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T01:12:21.221 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-15T01:12:21.221 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T01:12:21.346 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-15T01:12:21.346 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T01:12:21.347 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-15T01:12:21.357 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-15T01:12:21.645 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T01:12:21.659 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-15T01:12:21.945 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T01:12:21.946 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-15T01:12:21.958 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":1603,"stamp":"2024-09-15T01:12:20.903043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":460204,"kb_used_data":11376,"kb_used_omap":4,"kb_used_meta":448763,"kb_avail":282655316,"statfs":{"total":289910292480,"available":289439043584,"internally_reserved":0,"allocated":11649024,"data_stored":7349014,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":459533647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002226"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.189758+0000","last_change":"2024-09-15T00:34:34.578484+0000","last_active":"2024-09-15T01:11:50.189758+0000","last_peered":"2024-09-15T01:11:50.189758+0000","last_clean":"2024-09-15T01:11:50.189758+0000","last_became_active":"2024-09-15T00:34:32.575307+0000","last_became_peered":"2024-09-15T00:34:32.575307+0000","last_unstale":"2024-09-15T01:11:50.189758+0000","last_undegraded":"2024-09-15T01:11:50.189758+0000","last_fullsized":"2024-09-15T01:11:50.189758+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:04:44.235727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010984479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":940,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.192191+0000","last_change":"2024-09-15T00:34:34.578159+0000","last_active":"2024-09-15T01:11:50.192191+0000","last_peered":"2024-09-15T01:11:50.192191+0000","last_clean":"2024-09-15T01:11:50.192191+0000","last_became_active":"2024-09-15T00:34:32.576552+0000","last_became_peered":"2024-09-15T00:34:32.576552+0000","last_unstale":"2024-09-15T01:11:50.192191+0000","last_undegraded":"2024-09-15T01:11:50.192191+0000","last_fullsized":"2024-09-15T01:11:50.192191+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:30:22.053783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001076133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191453+0000","last_change":"2024-09-15T00:34:34.577770+0000","last_active":"2024-09-15T01:11:50.191453+0000","last_peered":"2024-09-15T01:11:50.191453+0000","last_clean":"2024-09-15T01:11:50.191453+0000","last_became_active":"2024-09-15T00:34:32.574697+0000","last_became_peered":"2024-09-15T00:34:32.574697+0000","last_unstale":"2024-09-15T01:11:50.191453+0000","last_undegraded":"2024-09-15T01:11:50.191453+0000","last_fullsized":"2024-09-15T01:11:50.191453+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:56:34.709213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073585800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191649+0000","last_change":"2024-09-15T00:34:34.578167+0000","last_active":"2024-09-15T01:11:50.191649+0000","last_peered":"2024-09-15T01:11:50.191649+0000","last_clean":"2024-09-15T01:11:50.191649+0000","last_became_active":"2024-09-15T00:34:32.575384+0000","last_became_peered":"2024-09-15T00:34:32.575384+0000","last_unstale":"2024-09-15T01:11:50.191649+0000","last_undegraded":"2024-09-15T01:11:50.191649+0000","last_fullsized":"2024-09-15T01:11:50.191649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:06:32.098451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010254979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":943,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.660903+0000","last_change":"2024-09-15T00:34:34.574756+0000","last_active":"2024-09-15T01:11:52.660903+0000","last_peered":"2024-09-15T01:11:52.660903+0000","last_clean":"2024-09-15T01:11:52.660903+0000","last_became_active":"2024-09-15T00:34:32.575639+0000","last_became_peered":"2024-09-15T00:34:32.575639+0000","last_unstale":"2024-09-15T01:11:52.660903+0000","last_undegraded":"2024-09-15T01:11:52.660903+0000","last_fullsized":"2024-09-15T01:11:52.660903+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:44:05.451379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054626299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.661130+0000","last_change":"2024-09-15T00:34:34.912915+0000","last_active":"2024-09-15T01:11:52.661130+0000","last_peered":"2024-09-15T01:11:52.661130+0000","last_clean":"2024-09-15T01:11:52.661130+0000","last_became_active":"2024-09-15T00:34:32.573975+0000","last_became_peered":"2024-09-15T00:34:32.573975+0000","last_unstale":"2024-09-15T01:11:52.661130+0000","last_undegraded":"2024-09-15T01:11:52.661130+0000","last_fullsized":"2024-09-15T01:11:52.661130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:42.456821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040528,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":941,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:52.662529+0000","last_change":"2024-09-15T00:34:34.912807+0000","last_active":"2024-09-15T01:11:52.662529+0000","last_peered":"2024-09-15T01:11:52.662529+0000","last_clean":"2024-09-15T01:11:52.662529+0000","last_became_active":"2024-09-15T00:34:32.573804+0000","last_became_peered":"2024-09-15T00:34:32.573804+0000","last_unstale":"2024-09-15T01:11:52.662529+0000","last_undegraded":"2024-09-15T01:11:52.662529+0000","last_fullsized":"2024-09-15T01:11:52.662529+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:29:40.374387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000336428,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":942,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.191278+0000","last_change":"2024-09-15T00:34:34.577734+0000","last_active":"2024-09-15T01:11:50.191278+0000","last_peered":"2024-09-15T01:11:50.191278+0000","last_clean":"2024-09-15T01:11:50.191278+0000","last_became_active":"2024-09-15T00:34:32.573483+0000","last_became_peered":"2024-09-15T00:34:32.573483+0000","last_unstale":"2024-09-15T01:11:50.191278+0000","last_undegraded":"2024-09-15T01:11:50.191278+0000","last_fullsized":"2024-09-15T01:11:50.191278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:31.560961+0000","last_clean_scrub_stamp":"2024-09-15T00:34:31.560961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:56:27.571672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067056699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1069,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:11:50.192800+0000","last_change":"2024-09-15T00:34:25.561004+0000","last_active":"2024-09-15T01:11:50.192800+0000","last_peered":"2024-09-15T01:11:50.192800+0000","last_clean":"2024-09-15T01:11:50.192800+0000","last_became_active":"2024-09-15T00:34:25.559814+0000","last_became_peered":"2024-09-15T00:34:25.559814+0000","last_unstale":"2024-09-15T01:11:50.192800+0000","last_undegraded":"2024-09-15T01:11:50.192800+0000","last_fullsized":"2024-09-15T01:11:50.192800+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T00:34:24.549251+0000","last_clean_scrub_stamp":"2024-09-15T00:34:24.549251+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:04:58.147756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706125,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165432,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206408,"statfs":{"total":96636764160,"available":96467361792,"internally_reserved":0,"allocated":3903488,"data_stored":2646467,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129276,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94242564,"statfs":{"total":96636764160,"available":96504385536,"internally_reserved":0,"allocated":3842048,"data_stored":2056080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738827,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165496,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206344,"statfs":{"total":96636764160,"available":96467296256,"internally_reserved":0,"allocated":3903488,"data_stored":2646467,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:12:21.959 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-15T00:34:31.560961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=34, tm_sec=31, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=12, tm_sec=19, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:12:21.959 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-15T00:34:31.560961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=34, tm_sec=31, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=12, tm_sec=19, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:12:21.959 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-15T00:34:31.560961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=34, tm_sec=31, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=12, tm_sec=19, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:12:21.959 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-15T00:34:31.560961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=34, tm_sec=31, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=12, tm_sec=19, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:12:21.959 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-15T00:34:31.560961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=34, tm_sec=31, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=12, tm_sec=19, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:12:21.959 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-15T00:34:31.560961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=34, tm_sec=31, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=12, tm_sec=19, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:12:21.959 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T00:34:31.560961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=34, tm_sec=31, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=12, tm_sec=19, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:12:21.960 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-15T00:34:31.560961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=34, tm_sec=31, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=12, tm_sec=19, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:12:21.960 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T00:34:24.549251+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=0, tm_min=34, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=12, tm_sec=19, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T01:12:21.960 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T01:12:41.961 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-15T01:12:42.262 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-15T01:12:42.262 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-15T01:12:42.274 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":1613,"stamp":"2024-09-15T01:12:40.907131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":460192,"kb_used_data":11376,"kb_used_omap":4,"kb_used_meta":448763,"kb_avail":282655328,"statfs":{"total":289910292480,"available":289439055872,"internally_reserved":0,"allocated":11649024,"data_stored":7349014,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":459533647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":949,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:12:25.174963+0000","last_change":"2024-09-15T01:12:25.175011+0000","last_active":"2024-09-15T01:12:25.174963+0000","last_peered":"2024-09-15T01:12:25.174963+0000","last_clean":"2024-09-15T01:12:25.174963+0000","last_became_active":"2024-09-15T00:34:32.575307+0000","last_became_peered":"2024-09-15T00:34:32.575307+0000","last_unstale":"2024-09-15T01:12:25.174963+0000","last_undegraded":"2024-09-15T01:12:25.174963+0000","last_fullsized":"2024-09-15T01:12:25.174963+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:12:25.174851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:12:25.174851+0000","last_clean_scrub_stamp":"2024-09-15T01:12:25.174851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:02.120071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010984479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:12:27.155332+0000","last_change":"2024-09-15T01:12:27.155359+0000","last_active":"2024-09-15T01:12:27.155332+0000","last_peered":"2024-09-15T01:12:27.155332+0000","last_clean":"2024-09-15T01:12:27.155332+0000","last_became_active":"2024-09-15T00:34:32.576552+0000","last_became_peered":"2024-09-15T00:34:32.576552+0000","last_unstale":"2024-09-15T01:12:27.155332+0000","last_undegraded":"2024-09-15T01:12:27.155332+0000","last_fullsized":"2024-09-15T01:12:27.155332+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:12:27.155228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:12:27.155228+0000","last_clean_scrub_stamp":"2024-09-15T01:12:27.155228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:41:07.968751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001076133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":950,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:12:24.152733+0000","last_change":"2024-09-15T01:12:24.152758+0000","last_active":"2024-09-15T01:12:24.152733+0000","last_peered":"2024-09-15T01:12:24.152733+0000","last_clean":"2024-09-15T01:12:24.152733+0000","last_became_active":"2024-09-15T00:34:32.574697+0000","last_became_peered":"2024-09-15T00:34:32.574697+0000","last_unstale":"2024-09-15T01:12:24.152733+0000","last_undegraded":"2024-09-15T01:12:24.152733+0000","last_fullsized":"2024-09-15T01:12:24.152733+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:12:24.152609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:12:24.152609+0000","last_clean_scrub_stamp":"2024-09-15T01:12:24.152609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:13:37.388872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073585800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":950,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:12:26.174775+0000","last_change":"2024-09-15T01:12:26.174825+0000","last_active":"2024-09-15T01:12:26.174775+0000","last_peered":"2024-09-15T01:12:26.174775+0000","last_clean":"2024-09-15T01:12:26.174775+0000","last_became_active":"2024-09-15T00:34:32.575384+0000","last_became_peered":"2024-09-15T00:34:32.575384+0000","last_unstale":"2024-09-15T01:12:26.174775+0000","last_undegraded":"2024-09-15T01:12:26.174775+0000","last_fullsized":"2024-09-15T01:12:26.174775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:12:26.174635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:12:26.174635+0000","last_clean_scrub_stamp":"2024-09-15T01:12:26.174635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:50:43.867251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010254979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"467'3","reported_seq":953,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:12:21.482793+0000","last_change":"2024-09-15T01:12:21.482817+0000","last_active":"2024-09-15T01:12:21.482793+0000","last_peered":"2024-09-15T01:12:21.482793+0000","last_clean":"2024-09-15T01:12:21.482793+0000","last_became_active":"2024-09-15T00:34:32.575639+0000","last_became_peered":"2024-09-15T00:34:32.575639+0000","last_unstale":"2024-09-15T01:12:21.482793+0000","last_undegraded":"2024-09-15T01:12:21.482793+0000","last_fullsized":"2024-09-15T01:12:21.482793+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'3","last_scrub_stamp":"2024-09-15T01:12:21.482714+0000","last_deep_scrub":"467'3","last_deep_scrub_stamp":"2024-09-15T01:12:21.482714+0000","last_clean_scrub_stamp":"2024-09-15T01:12:21.482714+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:06.486615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054626299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":950,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:12:21.922975+0000","last_change":"2024-09-15T01:12:21.922998+0000","last_active":"2024-09-15T01:12:21.922975+0000","last_peered":"2024-09-15T01:12:21.922975+0000","last_clean":"2024-09-15T01:12:21.922975+0000","last_became_active":"2024-09-15T00:34:32.573975+0000","last_became_peered":"2024-09-15T00:34:32.573975+0000","last_unstale":"2024-09-15T01:12:21.922975+0000","last_undegraded":"2024-09-15T01:12:21.922975+0000","last_fullsized":"2024-09-15T01:12:21.922975+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:12:21.922894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:12:21.922894+0000","last_clean_scrub_stamp":"2024-09-15T01:12:21.922894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:33:40.590744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040528,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":951,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:12:22.934405+0000","last_change":"2024-09-15T01:12:22.934405+0000","last_active":"2024-09-15T01:12:22.934405+0000","last_peered":"2024-09-15T01:12:22.934405+0000","last_clean":"2024-09-15T01:12:22.934405+0000","last_became_active":"2024-09-15T00:34:32.573804+0000","last_became_peered":"2024-09-15T00:34:32.573804+0000","last_unstale":"2024-09-15T01:12:22.934405+0000","last_undegraded":"2024-09-15T01:12:22.934405+0000","last_fullsized":"2024-09-15T01:12:22.934405+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:12:22.934274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:12:22.934274+0000","last_clean_scrub_stamp":"2024-09-15T01:12:22.934274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:32:45.657507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000336428,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"467'2","reported_seq":952,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:12:23.116254+0000","last_change":"2024-09-15T01:12:23.116285+0000","last_active":"2024-09-15T01:12:23.116254+0000","last_peered":"2024-09-15T01:12:23.116254+0000","last_clean":"2024-09-15T01:12:23.116254+0000","last_became_active":"2024-09-15T00:34:32.573483+0000","last_became_peered":"2024-09-15T00:34:32.573483+0000","last_unstale":"2024-09-15T01:12:23.116254+0000","last_undegraded":"2024-09-15T01:12:23.116254+0000","last_fullsized":"2024-09-15T01:12:23.116254+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'2","last_scrub_stamp":"2024-09-15T01:12:23.116164+0000","last_deep_scrub":"467'2","last_deep_scrub_stamp":"2024-09-15T01:12:23.116164+0000","last_clean_scrub_stamp":"2024-09-15T01:12:23.116164+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:43.374340+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00067056699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"467'99","reported_seq":1081,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-15T01:12:22.113864+0000","last_change":"2024-09-15T01:12:22.113864+0000","last_active":"2024-09-15T01:12:22.113864+0000","last_peered":"2024-09-15T01:12:22.113864+0000","last_clean":"2024-09-15T01:12:22.113864+0000","last_became_active":"2024-09-15T00:34:25.559814+0000","last_became_peered":"2024-09-15T00:34:25.559814+0000","last_unstale":"2024-09-15T01:12:22.113864+0000","last_undegraded":"2024-09-15T01:12:22.113864+0000","last_fullsized":"2024-09-15T01:12:22.113864+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'99","last_scrub_stamp":"2024-09-15T01:12:22.113762+0000","last_deep_scrub":"467'99","last_deep_scrub_stamp":"2024-09-15T01:12:22.113762+0000","last_clean_scrub_stamp":"2024-09-15T01:12:22.113762+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:13:41.145963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11506,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706129,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165428,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206412,"statfs":{"total":96636764160,"available":96467365888,"internally_reserved":0,"allocated":3903488,"data_stored":2646467,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129272,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94242568,"statfs":{"total":96636764160,"available":96504389632,"internally_reserved":0,"allocated":3842048,"data_stored":2056080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165492,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206348,"statfs":{"total":96636764160,"available":96467300352,"internally_reserved":0,"allocated":3903488,"data_stored":2646467,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5753,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:12:42.276 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-15T01:12:42.578 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T01:12:42.579 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T01:12:42.579 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-15T01:12:42.579 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:12:42.626 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-15T01:12:42.626 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-15T01:12:42.626 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:12:42.672 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-15T01:12:42.672 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-15T01:12:42.672 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:12:42.730 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-15T01:12:42.730 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T01:12:42.730 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-15T01:12:42.731 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:12:42.757 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-15T01:12:42.757 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T01:12:42.757 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-15T01:12:42.757 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:12:42.810 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-15T01:12:42.810 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T01:12:42.810 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-15T01:12:42.878 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-15T01:12:42.878 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-15T01:12:43.074 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-15T01:12:43.074 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-15T01:12:43.162 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-15T01:12:43.163 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-15T01:12:43.254 INFO:tasks.ceph:Archiving mon data... 2024-09-15T01:12:43.254 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845/data/cluster1.mon.a.tgz 2024-09-15T01:12:43.254 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-09-15T01:12:43.271 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.JuAVqWnCXd 2024-09-15T01:12:43.271 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.JuAVqWnCXd 2024-09-15T01:12:43.437 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.JuAVqWnCXd 2024-09-15T01:12:43.521 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.JuAVqWnCXd is 421KB 2024-09-15T01:12:43.617 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.JuAVqWnCXd 2024-09-15T01:12:43.633 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T01:12:43.633 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T01:12:43.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T01:12:43.754 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-15T01:12:43.776 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-15T01:12:43.776 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T01:12:43.778 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845/remote/smithi002/crash 2024-09-15T01:12:43.779 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T01:12:43.820 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845/remote/smithi059/crash 2024-09-15T01:12:43.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T01:12:43.851 INFO:tasks.ceph:Compressing logs... 2024-09-15T01:12:43.852 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T01:12:43.865 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T01:12:43.910 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35424.log 2024-09-15T01:12:43.910 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37005.log 2024-09-15T01:12:43.910 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5/var/log/ceph/cluster1-client.admin.35424.log: --verbose -- /var/log/ceph/cluster1-client.admin.35423.log 2024-09-15T01:12:43.910 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35424.log.gz 2024-09-15T01:12:43.910 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.37005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-15T01:12:43.910 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37005.log.gz 2024-09-15T01:12:43.910 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35720.log 2024-09-15T01:12:43.910 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35423.log.gz 2024-09-15T01:12:43.910 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35297.log 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32061.log 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35720.log.gz 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35296.log 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35297.log.gz 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.32061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32101.log 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32061.log.gz 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35384.log 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35296.log.gz 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.32101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35132.log 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32101.log.gz 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35056.log 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35384.log.gz 2024-09-15T01:12:43.911 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35510.log 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35132.log.gz 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35056.log.gz 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35298.log 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35510.log.gz 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32021.log 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35298.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.35298.log.gz 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.36627.log 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.32021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37047.log 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32021.log.gz 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.36627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36669.log 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36627.log.gz 2024-09-15T01:12:43.912 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.37047.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35174.log 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37047.log.gz 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.36669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36669.log.gz 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37129.log 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35174.log.gz 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36963.log 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.37129.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27371.log 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37129.log.gz 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.36963.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/cluster1-client.admin.36963.log.gz -- 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr: /var/log/ceph/cluster2-mgr.x.log 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35552.log 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.tmp-client.admin.27371.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27371.log.gz 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35216.log 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36753.log 2024-09-15T01:12:43.913 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35552.log.gz 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36921.log 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35216.log.gz 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.36753.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36753.log.gz 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.36921.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36921.log.gz 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32200.log 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37089.log 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.32200.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32200.log.gz 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36711.log 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.37089.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37089.log.gz 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31863.log 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.36711.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36711.log.gz 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-15T01:12:43.914 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.31863.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31863.log.gz 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.35636.log 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34916.log 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35636.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35636.log.gz 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35000.log 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.34916.log: /var/log/ceph/cluster1-osd.0.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34916.log.gz 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36879.log 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35000.log.gz 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:gzip 93.0% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.36795.log 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/cluster1-client.admin.36879.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.35678.log 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36879.log.gz 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.36795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36837.log 2024-09-15T01:12:43.915 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0%/var/log/ceph/cluster1-client.admin.35678.log: -- replaced with /var/log/ceph/cluster1-client.admin.36795.log.gz 2024-09-15T01:12:43.916 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35678.log.gz 2024-09-15T01:12:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35530.log 2024-09-15T01:12:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.36837.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36837.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32275.log 2024-09-15T01:12:43.916 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-15T01:12:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35081.log 2024-09-15T01:12:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.32275.log: /var/log/ceph/cluster1-client.admin.35530.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35530.log.gz 25.5% -- replaced with /var/log/ceph/cluster1-client.admin.32275.log.gz 2024-09-15T01:12:43.916 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-15T01:12:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-15T01:12:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1-client.admin.35081.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.35081.log.gz 2024-09-15T01:12:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cluster1.log: 87.4% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-15T01:12:43.943 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50237.log 2024-09-15T01:12:43.943 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58638.log 2024-09-15T01:12:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60458.log 2024-09-15T01:12:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56410.log 2024-09-15T01:12:43.945 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50237.log.gz 2024-09-15T01:12:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60458.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58638.log.gz 2024-09-15T01:12:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44370.log 2024-09-15T01:12:43.946 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60458.log.gz 2024-09-15T01:12:43.946 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51611.log 2024-09-15T01:12:43.946 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44370.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56410.log.gz 2024-09-15T01:12:43.946 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51137.log 2024-09-15T01:12:43.947 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50981.log 2024-09-15T01:12:43.947 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51137.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65262.log 2024-09-15T01:12:43.947 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37405.log 2024-09-15T01:12:43.947 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62984.log 2024-09-15T01:12:43.947 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67554.log 2024-09-15T01:12:43.947 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.48168.log 2024-09-15T01:12:43.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56994.log 2024-09-15T01:12:43.950 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62984.log.gz 2024-09-15T01:12:43.950 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67554.log.gz 2024-09-15T01:12:43.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.0.48168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57625.log 2024-09-15T01:12:43.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65532.log 2024-09-15T01:12:43.950 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65262.log.gz 2024-09-15T01:12:43.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47687.log 2024-09-15T01:12:43.950 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56994.log.gz 2024-09-15T01:12:43.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48479.log 2024-09-15T01:12:43.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64640.log 2024-09-15T01:12:43.951 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65532.log.gz 2024-09-15T01:12:43.951 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60595.log 2024-09-15T01:12:43.951 INFO:teuthology.orchestra.run.smithi059.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57625.log.gz 2024-09-15T01:12:43.951 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64640.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48640.log 2024-09-15T01:12:43.954 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51113.log 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64640.log.gz 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48640.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45314.log 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69345.log 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68304.log 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60595.log.gz 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49115.log 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69345.log: /var/log/ceph/cluster1-client.mirror.68304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58229.log 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69345.log.gz 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68304.log.gz 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67622.log 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49115.log.gz 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44433.log 2024-09-15T01:12:43.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60173.log 2024-09-15T01:12:43.956 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67622.log.gz 2024-09-15T01:12:43.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65635.log 2024-09-15T01:12:43.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62321.log 2024-09-15T01:12:43.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65635.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58229.log.gz 2024-09-15T01:12:43.956 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68348.log 2024-09-15T01:12:43.956 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60173.log.gz 2024-09-15T01:12:43.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66497.log 2024-09-15T01:12:43.959 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65076.log 2024-09-15T01:12:43.959 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66497.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62321.log.gz 2024-09-15T01:12:43.959 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68158.log 2024-09-15T01:12:43.959 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66497.log.gz 2024-09-15T01:12:43.959 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71057.log 2024-09-15T01:12:43.959 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65076.log: /var/log/ceph/cluster2-client.mirror.68158.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65635.log.gz 2024-09-15T01:12:43.959 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51611.log.gz 2024-09-15T01:12:43.959 INFO:teuthology.orchestra.run.smithi059.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68348.log.gz 2024-09-15T01:12:43.959 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% 87.4% 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45314.log.gz 2024-09-15T01:12:43.960 INFO:teuthology.orchestra.run.smithi059.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68158.log.gz 2024-09-15T01:12:43.960 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65076.log.gz 2024-09-15T01:12:43.960 INFO:teuthology.orchestra.run.smithi059.stderr: 54.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37405.log.gz 2024-09-15T01:12:43.960 INFO:teuthology.orchestra.run.smithi059.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.48168.log.gz 2024-09-15T01:12:43.962 INFO:teuthology.orchestra.run.smithi059.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50981.log.gz 2024-09-15T01:12:43.962 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51113.log.gz 2024-09-15T01:12:43.962 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51137.log.gz 2024-09-15T01:12:43.962 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47687.log.gz 2024-09-15T01:12:43.962 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60127.log 2024-09-15T01:12:43.963 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43028.log 2024-09-15T01:12:43.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71057.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71057.log.gz 2024-09-15T01:12:43.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42290.log 2024-09-15T01:12:43.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.43028.log: gzip -5 --verbose -- 89.7% /var/log/ceph/cluster2-client.mirror.69424.log -- replaced with /var/log/ceph/cluster2-client.mirror.60127.log.gz 2024-09-15T01:12:43.963 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:43.963 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.43028.log.gz 2024-09-15T01:12:43.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42290.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44370.log.gz 2024-09-15T01:12:43.963 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50663.log 2024-09-15T01:12:43.963 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42290.log.gz 2024-09-15T01:12:43.963 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65801.log 2024-09-15T01:12:43.968 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69424.log: /var/log/ceph/cluster1-client.mirror.50663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64578.log 2024-09-15T01:12:43.968 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50663.log.gz 2024-09-15T01:12:43.968 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69424.log.gz 2024-09-15T01:12:43.968 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61181.log 2024-09-15T01:12:43.968 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64994.log 2024-09-15T01:12:43.968 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64578.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65801.log.gz 2024-09-15T01:12:43.968 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68910.log 2024-09-15T01:12:43.968 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61181.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64578.log.gz 2024-09-15T01:12:43.968 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64994.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61181.log.gz 2024-09-15T01:12:43.968 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65615.log 2024-09-15T01:12:43.969 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64994.log.gz 2024-09-15T01:12:43.969 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64226.log 2024-09-15T01:12:43.969 INFO:teuthology.orchestra.run.smithi059.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68910.log.gz 2024-09-15T01:12:43.969 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52761.log 2024-09-15T01:12:43.969 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65615.log: /var/log/ceph/cluster2-client.mirror.64226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58913.log 2024-09-15T01:12:43.969 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65615.log.gz 2024-09-15T01:12:43.969 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58546.log 2024-09-15T01:12:43.969 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52761.log.gz 2024-09-15T01:12:43.969 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53790.log 2024-09-15T01:12:43.969 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64371.log 2024-09-15T01:12:43.970 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58913.log.gz/var/log/ceph/cluster2-client.mirror.58546.log: 2024-09-15T01:12:43.970 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42007.log 2024-09-15T01:12:43.970 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41696.log 2024-09-15T01:12:43.970 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58546.log.gz 2024-09-15T01:12:43.970 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64226.log.gz 2024-09-15T01:12:43.970 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.42007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37251.log 2024-09-15T01:12:43.970 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44433.log.gz 2024-09-15T01:12:43.982 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64371.log.gz 2024-09-15T01:12:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41696.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42007.log.gz 2024-09-15T01:12:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66857.log 2024-09-15T01:12:43.982 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53790.log.gz 2024-09-15T01:12:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59968.log 2024-09-15T01:12:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.37251.log: /var/log/ceph/cluster1-client.mirror.66857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56885.log 2024-09-15T01:12:43.982 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66857.log.gz 2024-09-15T01:12:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54493.log 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59968.log.gz 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41510.log 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56885.log: /var/log/ceph/cluster2-client.mirror.54493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60779.log 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56885.log.gz 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.41510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38518.log 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60779.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54493.log.gz 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58863.log 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60779.log.gz 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39494.log 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.38518.log: /var/log/ceph/cluster1-client.mirror.58863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53613.log 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39494.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58863.log.gz 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44816.log 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38518.log.gz 2024-09-15T01:12:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51836.log 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44816.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53613.log.gz 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55011.log 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41696.log.gz 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63875.log 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51836.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44816.log.gz 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55011.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51836.log.gz 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62632.log 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63875.log: 87.6%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.55011.log.gz 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45356.log 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59138.log 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63875.log.gz 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45356.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67307.log 2024-09-15T01:12:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59138.log: gzip -5 --verbose 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37251.log.gz 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/cluster2-client.mirror.45490.log 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59138.log.gz 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.69211.log 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67307.log.gz 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58413.log 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45490.log.gz 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.0.69211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35450.log 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62632.log.gz 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52824.log 87.7% 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58413.log.gz 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35450.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35450.log.gz 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46837.log 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45895.log 2024-09-15T01:12:43.985 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50864.log 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52824.log.gz 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39494.log.gz 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45895.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.69211.log.gz 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42426.log 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45895.log.gz 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50864.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54447.log 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50864.log.gz/var/log/ceph/cluster2-client.mirror.42426.log: 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70886.log 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67329.log 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69974.log 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54447.log.gz 2024-09-15T01:12:43.986 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70886.log.gz 2024-09-15T01:12:44.008 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67329.log: gzip -5 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.67329.log.gz 2024-09-15T01:12:44.008 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.44616.log 2024-09-15T01:12:44.008 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69974.log: gzip 86.8% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57605.log 2024-09-15T01:12:44.008 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69974.log.gz 2024-09-15T01:12:44.008 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44716.log 2024-09-15T01:12:44.008 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.57605.log: -- /var/log/ceph/cluster2-client.mirror.46129.log 2024-09-15T01:12:44.008 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44616.log.gz 2024-09-15T01:12:44.008 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44716.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44716.log.gz 2024-09-15T01:12:44.008 INFO:teuthology.orchestra.run.smithi059.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57605.log.gz 2024-09-15T01:12:44.008 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38819.log 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42426.log.gz 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.41027.log 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46129.log: gzip/var/log/ceph/cluster1-client.mirror.38819.log: -5 --verbose 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46129.log.gz 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/cluster1-client.admin.37108.log 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.0.41027.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39758.log 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.37108.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37108.log.gz 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44069.log 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47356.log 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.40921.log 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47356.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41510.log.gz 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.3.40921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52512.log 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47356.log.gz 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67959.log 2024-09-15T01:12:44.009 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53364.log 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46512.log 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52512.log.gz 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9%/var/log/ceph/cluster1-client.mirror.53364.log: -- replaced with /var/log/ceph/cluster1-client.mirror.67959.log.gz 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54654.log 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66722.log 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53364.log.gz 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38819.log.gz 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54654.log: /var/log/ceph/cluster2-client.mirror.46512.log: gzip 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46512.log.gz 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62275.log 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35826.log 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62275.log: /var/log/ceph/cluster1-client.mirror.66722.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44069.log.gz 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr: 92.5% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66722.log.gz 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54654.log.gz 2024-09-15T01:12:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44696.log 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62275.log.gz 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35826.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71377.log 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.35826.log.gz 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38747.log 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71377.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49433.log 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71377.log.gz 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.38747.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.39758.log.gz 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/cluster1-client.mirror.66204.log 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44696.log.gz 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49433.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49433.log.gz 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44899.log 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/cluster1-client.mirror.66204.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69831.log 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44899.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66204.log.gz 2024-09-15T01:12:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58662.log 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44899.log.gz 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51185.log 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69831.log.gz 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58140.log 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58662.log.gz 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56294.log 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58140.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51185.log.gz 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46209.log 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58140.log.gz 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43253.log 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/cluster2-client.mirror.46209.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63978.log 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43253.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46209.log.gz 2024-09-15T01:12:44.012 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56294.log.gz 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40303.log 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.36862.log 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63978.log: /var/log/ceph/cluster2-client.mirror.40303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55742.log 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.3.36862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67419.log 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67757.log 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67419.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67419.log.gz 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66069.log 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67757.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55742.log.gz 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67757.log.gz 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47398.log 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63978.log.gz 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70484.log 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9%/var/log/ceph/cluster2-client.mirror.47398.log: -- replaced with /var/log/ceph/cluster1-client.mirror.66069.log.gz 2024-09-15T01:12:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41826.log 2024-09-15T01:12:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45170.log 2024-09-15T01:12:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70484.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43253.log.gz 2024-09-15T01:12:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72215.log 2024-09-15T01:12:44.017 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45170.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38747.log.gz 2024-09-15T01:12:44.020 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55509.log 2024-09-15T01:12:44.021 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72215.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72215.log.gz 2024-09-15T01:12:44.035 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48029.log 2024-09-15T01:12:44.036 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55509.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55509.log.gz 2024-09-15T01:12:44.038 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.36862.log.gz 2024-09-15T01:12:44.048 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40303.log.gz 2024-09-15T01:12:44.048 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41826.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47907.log 2024-09-15T01:12:44.049 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48029.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48029.log.gz 2024-09-15T01:12:44.057 INFO:teuthology.orchestra.run.smithi059.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41826.log.gz 2024-09-15T01:12:44.064 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54882.log 2024-09-15T01:12:44.064 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47907.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47907.log.gz 2024-09-15T01:12:44.072 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65979.log 2024-09-15T01:12:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54882.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54882.log.gz 2024-09-15T01:12:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.65979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59564.log 2024-09-15T01:12:44.082 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65979.log.gz 2024-09-15T01:12:44.099 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49945.log 2024-09-15T01:12:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59564.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59564.log.gz 2024-09-15T01:12:44.108 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64164.log 2024-09-15T01:12:44.121 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.49945.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49945.log.gz 2024-09-15T01:12:44.123 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70666.log 2024-09-15T01:12:44.126 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64164.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64164.log.gz 2024-09-15T01:12:44.132 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41784.log 2024-09-15T01:12:44.146 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70666.log: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.40921.log.gz 2024-09-15T01:12:44.147 INFO:teuthology.orchestra.run.smithi059.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70666.log.gz 2024-09-15T01:12:44.156 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.41027.log.gz 2024-09-15T01:12:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53049.log 2024-09-15T01:12:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47158.log 2024-09-15T01:12:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53049.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53049.log.gz 2024-09-15T01:12:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64598.log 2024-09-15T01:12:44.176 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53910.log 2024-09-15T01:12:44.177 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64598.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41784.log.gz 2024-09-15T01:12:44.178 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64598.log.gz 2024-09-15T01:12:44.189 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64392.log 2024-09-15T01:12:44.190 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53910.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53910.log.gz 2024-09-15T01:12:44.210 INFO:teuthology.orchestra.run.smithi002.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-15T01:12:44.214 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50960.log 2024-09-15T01:12:44.214 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64392.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64392.log.gz 2024-09-15T01:12:44.214 INFO:teuthology.orchestra.run.smithi002.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-15T01:12:44.240 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64185.log 2024-09-15T01:12:44.240 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50960.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50960.log.gz 2024-09-15T01:12:44.254 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50189.log 2024-09-15T01:12:44.254 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64185.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64185.log.gz 2024-09-15T01:12:44.269 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66362.log 2024-09-15T01:12:44.269 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50189.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50189.log.gz 2024-09-15T01:12:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.36582.log 2024-09-15T01:12:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66362.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66362.log.gz 2024-09-15T01:12:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50752.log 2024-09-15T01:12:44.306 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.0.36582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41278.log 2024-09-15T01:12:44.307 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50752.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50752.log.gz 2024-09-15T01:12:44.317 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45771.log 2024-09-15T01:12:44.321 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41278.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48640.log.gz 2024-09-15T01:12:44.322 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41278.log.gz 2024-09-15T01:12:44.323 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42266.log 2024-09-15T01:12:44.323 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45771.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45771.log.gz 2024-09-15T01:12:44.324 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.45993.log 2024-09-15T01:12:44.325 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42266.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42266.log.gz 2024-09-15T01:12:44.325 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.36582.log.gz 2024-09-15T01:12:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72509.log 2024-09-15T01:12:44.330 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.0.45993.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.45993.log.gz 2024-09-15T01:12:44.330 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62092.log 2024-09-15T01:12:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72509.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72509.log.gz 2024-09-15T01:12:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70162.log 2024-09-15T01:12:44.332 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62092.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62092.log.gz 2024-09-15T01:12:44.333 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43877.log 2024-09-15T01:12:44.333 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70162.log: 85.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70162.log.gz 2024-09-15T01:12:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50258.log 2024-09-15T01:12:44.340 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43877.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43877.log.gz 2024-09-15T01:12:44.341 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46894.log 2024-09-15T01:12:44.342 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50258.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50258.log.gz 2024-09-15T01:12:44.343 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42755.log 2024-09-15T01:12:44.345 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46894.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46894.log.gz 2024-09-15T01:12:44.345 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44175.log 2024-09-15T01:12:44.346 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42755.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42755.log.gz 2024-09-15T01:12:44.346 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55284.log 2024-09-15T01:12:44.352 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44175.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44175.log.gz 2024-09-15T01:12:44.353 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60866.log 2024-09-15T01:12:44.353 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55284.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55284.log.gz 2024-09-15T01:12:44.354 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60265.log 2024-09-15T01:12:44.356 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60866.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60866.log.gz 2024-09-15T01:12:44.356 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57263.log 2024-09-15T01:12:44.357 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60265.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60265.log.gz 2024-09-15T01:12:44.357 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.61850.log 2024-09-15T01:12:44.358 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57263.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57263.log.gz 2024-09-15T01:12:44.359 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48559.log 2024-09-15T01:12:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.1.61850.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.61850.log.gz 2024-09-15T01:12:44.365 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61069.log 2024-09-15T01:12:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37490.log 2024-09-15T01:12:44.383 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61069.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61069.log.gz 2024-09-15T01:12:44.385 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70484.log.gz 2024-09-15T01:12:44.385 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56069.log 2024-09-15T01:12:44.386 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37490.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37490.log.gz 2024-09-15T01:12:44.387 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69444.log 2024-09-15T01:12:44.387 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56069.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56069.log.gz 2024-09-15T01:12:44.389 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42935.log 2024-09-15T01:12:44.389 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69444.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69444.log.gz 2024-09-15T01:12:44.390 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39888.log 2024-09-15T01:12:44.395 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42935.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42935.log.gz 2024-09-15T01:12:44.395 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55083.log 2024-09-15T01:12:44.397 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39888.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39888.log.gz 2024-09-15T01:12:44.398 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47865.log 2024-09-15T01:12:44.399 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55083.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55083.log.gz 2024-09-15T01:12:44.399 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72467.log 2024-09-15T01:12:44.399 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47865.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47865.log.gz 2024-09-15T01:12:44.401 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62026.log 2024-09-15T01:12:44.401 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72467.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72467.log.gz 2024-09-15T01:12:44.402 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70384.log 2024-09-15T01:12:44.403 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62026.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62026.log.gz 2024-09-15T01:12:44.403 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68947.log 2024-09-15T01:12:44.404 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70384.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70384.log.gz 2024-09-15T01:12:44.405 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57062.log 2024-09-15T01:12:44.406 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68947.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68947.log.gz 2024-09-15T01:12:44.406 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52557.log 2024-09-15T01:12:44.408 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57062.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57062.log.gz 2024-09-15T01:12:44.408 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58209.log 2024-09-15T01:12:44.410 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52557.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52557.log.gz 2024-09-15T01:12:44.410 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44516.log 2024-09-15T01:12:44.413 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58209.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58209.log.gz 2024-09-15T01:12:44.413 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51635.log 2024-09-15T01:12:44.414 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44516.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44516.log.gz 2024-09-15T01:12:44.415 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50278.log 2024-09-15T01:12:44.416 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51635.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51635.log.gz 2024-09-15T01:12:44.416 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59183.log 2024-09-15T01:12:44.418 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50278.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50278.log.gz 2024-09-15T01:12:44.419 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35786.log 2024-09-15T01:12:44.422 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59183.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59183.log.gz 2024-09-15T01:12:44.422 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67104.log 2024-09-15T01:12:44.422 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35786.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35786.log.gz 2024-09-15T01:12:44.422 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45170.log.gz 2024-09-15T01:12:44.422 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43725.log 2024-09-15T01:12:44.423 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54561.log 2024-09-15T01:12:44.424 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67104.log: /var/log/ceph/cluster1-client.mirror.43725.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67104.log.gz 2024-09-15T01:12:44.424 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38584.log 2024-09-15T01:12:44.425 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54561.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54561.log.gz 2024-09-15T01:12:44.425 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69995.log 2024-09-15T01:12:44.426 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.38584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63128.log 2024-09-15T01:12:44.426 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38584.log.gz 2024-09-15T01:12:44.428 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69995.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69995.log.gz 2024-09-15T01:12:44.428 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39642.log 2024-09-15T01:12:44.429 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63128.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43725.log.gz 2024-09-15T01:12:44.429 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48320.log 2024-09-15T01:12:44.431 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39642.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39642.log.gz 2024-09-15T01:12:44.431 INFO:teuthology.orchestra.run.smithi059.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63128.log.gz 2024-09-15T01:12:44.431 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40609.log 2024-09-15T01:12:44.431 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46874.log 2024-09-15T01:12:44.432 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49492.log 2024-09-15T01:12:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46874.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46874.log.gz 2024-09-15T01:12:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66182.log 2024-09-15T01:12:44.440 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49492.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40609.log.gz 2024-09-15T01:12:44.453 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65573.log 2024-09-15T01:12:44.454 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66182.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66182.log.gz 2024-09-15T01:12:44.454 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68049.log 2024-09-15T01:12:44.457 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65573.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65573.log.gz 2024-09-15T01:12:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48399.log 2024-09-15T01:12:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68049.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68049.log.gz 2024-09-15T01:12:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58317.log 2024-09-15T01:12:44.490 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63190.log 2024-09-15T01:12:44.492 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58317.log: 87.7% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47158.log.gz 2024-09-15T01:12:44.493 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52374.log 2024-09-15T01:12:44.496 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63190.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63190.log.gz 2024-09-15T01:12:44.496 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37084.log 2024-09-15T01:12:44.498 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52374.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52374.log.gz 2024-09-15T01:12:44.498 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42567.log 2024-09-15T01:12:44.499 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.37084.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37084.log.gz 2024-09-15T01:12:44.499 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71626.log 2024-09-15T01:12:44.501 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42567.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42567.log.gz 2024-09-15T01:12:44.502 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44347.log 2024-09-15T01:12:44.502 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71626.log: 82.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71626.log.gz 2024-09-15T01:12:44.503 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56813.log 2024-09-15T01:12:44.504 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44347.log: -- replaced with /var/log/ceph/cluster2-client.mirror.58317.log.gz 2024-09-15T01:12:44.508 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44347.log.gz 2024-09-15T01:12:44.509 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67014.log 2024-09-15T01:12:44.509 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56813.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56813.log.gz 2024-09-15T01:12:44.510 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39993.log 2024-09-15T01:12:44.510 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67014.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67014.log.gz 2024-09-15T01:12:44.511 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59588.log 2024-09-15T01:12:44.511 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36458.log 2024-09-15T01:12:44.513 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59588.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59588.log.gz 2024-09-15T01:12:44.516 INFO:teuthology.orchestra.run.smithi059.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39993.log.gz 2024-09-15T01:12:44.525 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39472.log 2024-09-15T01:12:44.525 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.36458.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36458.log.gz 2024-09-15T01:12:44.537 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67532.log 2024-09-15T01:12:44.550 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61785.log 2024-09-15T01:12:44.551 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4%/var/log/ceph/cluster1-client.mirror.67532.log: -- replaced with /var/log/ceph/cluster1-client.mirror.39472.log.gz 2024-09-15T01:12:44.551 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67532.log.gz 2024-09-15T01:12:44.569 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51272.log 2024-09-15T01:12:44.569 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61785.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61785.log.gz 2024-09-15T01:12:44.579 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46837.log.gz 2024-09-15T01:12:44.580 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61004.log 2024-09-15T01:12:44.581 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51272.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51272.log.gz 2024-09-15T01:12:44.583 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54268.log 2024-09-15T01:12:44.583 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61004.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61004.log.gz 2024-09-15T01:12:44.583 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62253.log 2024-09-15T01:12:44.584 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54268.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54268.log.gz 2024-09-15T01:12:44.585 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40499.log 2024-09-15T01:12:44.587 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.62253.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62253.log.gz 2024-09-15T01:12:44.587 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53931.log 2024-09-15T01:12:44.588 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44636.log 2024-09-15T01:12:44.589 INFO:teuthology.orchestra.run.smithi059.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40499.log.gz 2024-09-15T01:12:44.592 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53931.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53931.log.gz 2024-09-15T01:12:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46954.log 2024-09-15T01:12:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44636.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44636.log.gz 2024-09-15T01:12:44.594 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32409.log 2024-09-15T01:12:44.596 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46954.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46954.log.gz 2024-09-15T01:12:44.597 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41761.log 2024-09-15T01:12:44.597 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.32409.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32409.log.gz 2024-09-15T01:12:44.598 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.49026.log 2024-09-15T01:12:44.598 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55960.log 2024-09-15T01:12:44.608 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.0.49026.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41761.log.gz 2024-09-15T01:12:44.622 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65429.log 2024-09-15T01:12:44.622 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55960.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55960.log.gz 2024-09-15T01:12:44.635 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47398.log.gz 2024-09-15T01:12:44.635 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42827.log 2024-09-15T01:12:44.638 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70254.log 2024-09-15T01:12:44.638 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65429.log: /var/log/ceph/cluster1-client.mirror.42827.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65429.log.gz 2024-09-15T01:12:44.638 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42827.log.gz 2024-09-15T01:12:44.638 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58297.log 2024-09-15T01:12:44.639 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70254.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70254.log.gz 2024-09-15T01:12:44.639 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55380.log 2024-09-15T01:12:44.640 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58297.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58297.log.gz 2024-09-15T01:12:44.641 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42605.log 2024-09-15T01:12:44.642 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55380.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55380.log.gz 2024-09-15T01:12:44.643 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67284.log 2024-09-15T01:12:44.644 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42605.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42605.log.gz 2024-09-15T01:12:44.645 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53702.log 2024-09-15T01:12:44.645 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67284.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67284.log.gz 2024-09-15T01:12:44.646 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57151.log 2024-09-15T01:12:44.648 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53702.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53702.log.gz 2024-09-15T01:12:44.648 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46109.log 2024-09-15T01:12:44.650 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57151.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.49026.log.gz 2024-09-15T01:12:44.651 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46974.log 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57151.log.gz 2024-09-15T01:12:44.651 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:44.651 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57986.log 2024-09-15T01:12:44.651 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46109.log: /var/log/ceph/cluster2-client.mirror.46974.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46974.log.gz 2024-09-15T01:12:44.652 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46549.log 2024-09-15T01:12:44.652 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46109.log.gz 2024-09-15T01:12:44.653 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57986.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57986.log.gz 2024-09-15T01:12:44.653 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63273.log 2024-09-15T01:12:44.654 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59271.log 2024-09-15T01:12:44.654 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46549.log: /var/log/ceph/cluster2-client.mirror.63273.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46549.log.gz 2024-09-15T01:12:44.654 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70995.log 2024-09-15T01:12:44.655 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63273.log.gz 2024-09-15T01:12:44.655 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59271.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59271.log.gz 2024-09-15T01:12:44.656 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40473.log 2024-09-15T01:12:44.656 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66564.log 2024-09-15T01:12:44.657 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70995.log: /var/log/ceph/cluster1-client.admin.40473.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70995.log.gz 2024-09-15T01:12:44.657 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63170.log 2024-09-15T01:12:44.658 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65511.log 2024-09-15T01:12:44.658 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66564.log: /var/log/ceph/cluster2-client.mirror.63170.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66564.log.gz 2024-09-15T01:12:44.660 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66317.log 2024-09-15T01:12:44.660 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63170.log.gz 2024-09-15T01:12:44.660 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56948.log 2024-09-15T01:12:44.660 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66317.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66317.log.gz 2024-09-15T01:12:44.660 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48537.log 2024-09-15T01:12:44.664 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56948.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56948.log.gz 2024-09-15T01:12:44.664 INFO:teuthology.orchestra.run.smithi059.stderr: 90.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53232.log 2024-09-15T01:12:44.664 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65511.log.gz 2024-09-15T01:12:44.664 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48537.log: gzip 87.4% -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71169.log 2024-09-15T01:12:44.664 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48537.log.gz 2024-09-15T01:12:44.664 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45208.log 2024-09-15T01:12:44.664 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71169.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71169.log.gz 2024-09-15T01:12:44.664 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61536.log 2024-09-15T01:12:44.665 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45208.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53232.log.gz 2024-09-15T01:12:44.665 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65677.log 2024-09-15T01:12:44.668 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61536.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61536.log.gz 2024-09-15T01:12:44.668 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.40473.log.gz 2024-09-15T01:12:44.669 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36080.log 2024-09-15T01:12:44.669 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48479.log.gz 2024-09-15T01:12:44.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59722.log 2024-09-15T01:12:44.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.36080.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36080.log.gz 2024-09-15T01:12:44.669 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65677.log.gz 2024-09-15T01:12:44.669 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70646.log 2024-09-15T01:12:44.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46151.log 2024-09-15T01:12:44.670 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70646.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70646.log.gz 2024-09-15T01:12:44.671 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59722.log.gz 2024-09-15T01:12:44.671 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39574.log 2024-09-15T01:12:44.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53211.log 2024-09-15T01:12:44.672 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.39574.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39574.log.gz 2024-09-15T01:12:44.672 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69153.log 2024-09-15T01:12:44.673 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53211.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53211.log.gz 2024-09-15T01:12:44.673 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66137.log 2024-09-15T01:12:44.674 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69153.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69153.log.gz 2024-09-15T01:12:44.675 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41648.log 2024-09-15T01:12:44.676 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66137.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66137.log.gz 2024-09-15T01:12:44.676 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67172.log 2024-09-15T01:12:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41648.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41648.log.gz 2024-09-15T01:12:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44327.log 2024-09-15T01:12:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67172.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67172.log.gz 2024-09-15T01:12:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57785.log 2024-09-15T01:12:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44327.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44327.log.gz 2024-09-15T01:12:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43553.log 2024-09-15T01:12:44.686 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57785.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57785.log.gz 2024-09-15T01:12:44.687 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51338.log 2024-09-15T01:12:44.693 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43553.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43553.log.gz 2024-09-15T01:12:44.693 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49571.log 2024-09-15T01:12:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51338.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51338.log.gz 2024-09-15T01:12:44.695 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38860.log 2024-09-15T01:12:44.711 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49571.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59450.log 2024-09-15T01:12:44.716 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.38860.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49492.log.gz 2024-09-15T01:12:44.716 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62188.log 2024-09-15T01:12:44.718 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59450.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38860.log.gz 2024-09-15T01:12:44.718 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59450.log.gz 2024-09-15T01:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60912.log 2024-09-15T01:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.62188.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62188.log.gz 2024-09-15T01:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63066.log 2024-09-15T01:12:44.722 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60912.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60912.log.gz 2024-09-15T01:12:44.722 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59790.log 2024-09-15T01:12:44.725 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63066.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63066.log.gz 2024-09-15T01:12:44.725 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45631.log 2024-09-15T01:12:44.726 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59790.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59790.log.gz 2024-09-15T01:12:44.726 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65159.log 2024-09-15T01:12:44.746 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70930.log 2024-09-15T01:12:44.747 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65159.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65159.log.gz 2024-09-15T01:12:44.762 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64350.log 2024-09-15T01:12:44.763 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70930.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70930.log.gz 2024-09-15T01:12:44.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67667.log 2024-09-15T01:12:44.790 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64350.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64350.log.gz 2024-09-15T01:12:44.793 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45356.log.gz 2024-09-15T01:12:44.794 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37132.log 2024-09-15T01:12:44.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67667.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67667.log.gz 2024-09-15T01:12:44.795 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62549.log 2024-09-15T01:12:44.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.37132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60663.log 2024-09-15T01:12:44.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62549.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62549.log.gz 2024-09-15T01:12:44.797 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64952.log 2024-09-15T01:12:44.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60663.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60663.log.gz 2024-09-15T01:12:44.800 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62777.log 2024-09-15T01:12:44.803 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64952.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64952.log.gz 2024-09-15T01:12:44.803 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67487.log 2024-09-15T01:12:44.804 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62777.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62777.log.gz 2024-09-15T01:12:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51458.log 2024-09-15T01:12:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67487.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67487.log.gz 2024-09-15T01:12:44.806 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59947.log 2024-09-15T01:12:44.807 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51458.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51458.log.gz 2024-09-15T01:12:44.807 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49253.log 2024-09-15T01:12:44.809 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59947.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59947.log.gz 2024-09-15T01:12:44.809 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57444.log 2024-09-15T01:12:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49253.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.37132.log.gz 2024-09-15T01:12:44.825 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60575.log 2024-09-15T01:12:44.827 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57444.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57444.log.gz 2024-09-15T01:12:44.827 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52645.log 2024-09-15T01:12:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60575.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60575.log.gz 2024-09-15T01:12:44.842 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61761.log 2024-09-15T01:12:44.844 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52645.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52645.log.gz 2024-09-15T01:12:44.844 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61049.log 2024-09-15T01:12:44.845 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61761.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61761.log.gz 2024-09-15T01:12:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52440.log 2024-09-15T01:12:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61049.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61049.log.gz 2024-09-15T01:12:44.868 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57512.log 2024-09-15T01:12:44.869 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52440.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52440.log.gz 2024-09-15T01:12:44.871 INFO:teuthology.orchestra.run.smithi059.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45208.log.gz 2024-09-15T01:12:44.872 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37371.log 2024-09-15T01:12:44.873 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57512.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57512.log.gz 2024-09-15T01:12:44.873 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50525.log 2024-09-15T01:12:44.874 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.37371.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.37371.log.gz 2024-09-15T01:12:44.874 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64247.log 2024-09-15T01:12:44.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50525.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50525.log.gz 2024-09-15T01:12:44.876 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59992.log 2024-09-15T01:12:44.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64247.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64247.log.gz 2024-09-15T01:12:44.878 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50483.log 2024-09-15T01:12:44.879 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59992.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59992.log.gz 2024-09-15T01:12:44.879 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45751.log 2024-09-15T01:12:44.882 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50483.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50483.log.gz 2024-09-15T01:12:44.882 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43160.log 2024-09-15T01:12:44.884 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45751.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45751.log.gz 2024-09-15T01:12:44.884 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50141.log 2024-09-15T01:12:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43160.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43160.log.gz 2024-09-15T01:12:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.58051.log 2024-09-15T01:12:44.886 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50141.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50141.log.gz 2024-09-15T01:12:44.887 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57173.log 2024-09-15T01:12:44.891 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.0.58051.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.58051.log.gz 2024-09-15T01:12:44.892 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66159.log 2024-09-15T01:12:44.893 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57173.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57173.log.gz 2024-09-15T01:12:44.893 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56161.log 2024-09-15T01:12:44.894 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66159.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66159.log.gz 2024-09-15T01:12:44.895 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61335.log 2024-09-15T01:12:44.896 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56161.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56161.log.gz 2024-09-15T01:12:44.896 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45087.log 2024-09-15T01:12:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61335.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61335.log.gz 2024-09-15T01:12:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37149.log 2024-09-15T01:12:44.910 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46432.log 2024-09-15T01:12:44.914 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.37149.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48559.log.gz 2024-09-15T01:12:44.915 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49882.log 2024-09-15T01:12:44.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46432.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46432.log.gz 2024-09-15T01:12:44.920 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66227.log 2024-09-15T01:12:44.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49882.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.37149.log.gz 2024-09-15T01:12:44.920 INFO:teuthology.orchestra.run.smithi059.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49882.log.gz 2024-09-15T01:12:44.920 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35699.log 2024-09-15T01:12:44.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66227.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66227.log.gz 2024-09-15T01:12:44.920 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45572.log 2024-09-15T01:12:44.921 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35699.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35699.log.gz 2024-09-15T01:12:44.921 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37388.log 2024-09-15T01:12:44.921 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45572.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45572.log.gz 2024-09-15T01:12:44.921 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57854.log 2024-09-15T01:12:44.922 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.37388.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45028.log 2024-09-15T01:12:44.922 INFO:teuthology.orchestra.run.smithi059.stderr: 66.2% -- replaced with /var/log/ceph/cluster2-client.admin.37388.log.gz 2024-09-15T01:12:44.923 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57854.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57854.log.gz 2024-09-15T01:12:44.923 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37566.log 2024-09-15T01:12:44.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45028.log: 77.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45028.log.gz 2024-09-15T01:12:44.925 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51001.log 2024-09-15T01:12:44.926 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55557.log 2024-09-15T01:12:44.926 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37566.log.gz 2024-09-15T01:12:44.929 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51001.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51001.log.gz 2024-09-15T01:12:44.929 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69909.log 2024-09-15T01:12:44.930 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55557.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55557.log.gz 2024-09-15T01:12:44.930 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.45394.log 2024-09-15T01:12:44.932 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69909.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69909.log.gz 2024-09-15T01:12:44.932 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66969.log 2024-09-15T01:12:44.932 INFO:teuthology.orchestra.run.smithi002.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-15T01:12:44.935 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.2.45394.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.45394.log.gz 2024-09-15T01:12:44.935 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.71626.log 2024-09-15T01:12:44.937 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66969.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66969.log.gz 2024-09-15T01:12:44.937 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42136.log 2024-09-15T01:12:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.3.71626.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.71626.log.gz 2024-09-15T01:12:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55107.log 2024-09-15T01:12:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42136.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42136.log.gz 2024-09-15T01:12:44.940 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51318.log 2024-09-15T01:12:44.942 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55107.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55107.log.gz 2024-09-15T01:12:44.942 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58341.log 2024-09-15T01:12:44.943 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51318.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51318.log.gz 2024-09-15T01:12:44.943 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59879.log 2024-09-15T01:12:44.944 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58341.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58341.log.gz 2024-09-15T01:12:44.945 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60085.log 2024-09-15T01:12:44.948 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59879.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59879.log.gz 2024-09-15T01:12:44.948 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47540.log 2024-09-15T01:12:44.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60085.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60085.log.gz 2024-09-15T01:12:44.950 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66002.log 2024-09-15T01:12:44.951 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47540.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47540.log.gz 2024-09-15T01:12:44.952 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37328.log 2024-09-15T01:12:44.953 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66002.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66002.log.gz 2024-09-15T01:12:44.953 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40722.log 2024-09-15T01:12:44.954 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.37328.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37328.log.gz 2024-09-15T01:12:44.954 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38611.log 2024-09-15T01:12:44.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47520.log 2024-09-15T01:12:44.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.38611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40921.log 2024-09-15T01:12:44.957 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1%/var/log/ceph/cluster2-client.mirror.47520.log: -- replaced with /var/log/ceph/cluster1-client.admin.38611.log.gz 2024-09-15T01:12:44.957 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47520.log.gz 2024-09-15T01:12:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45148.log 2024-09-15T01:12:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52019.log 2024-09-15T01:12:44.959 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45148.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45148.log.gz 2024-09-15T01:12:44.959 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62653.log 2024-09-15T01:12:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52019.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49571.log.gz 2024-09-15T01:12:44.960 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52019.log.gz 2024-09-15T01:12:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57219.log 2024-09-15T01:12:44.962 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62653.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40921.log.gz 2024-09-15T01:12:44.962 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62653.log.gz 2024-09-15T01:12:44.962 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66767.log 2024-09-15T01:12:44.964 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57219.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57219.log.gz 2024-09-15T01:12:44.964 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66452.log 2024-09-15T01:12:44.965 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66767.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66767.log.gz 2024-09-15T01:12:44.965 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62612.log 2024-09-15T01:12:44.966 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66452.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66452.log.gz 2024-09-15T01:12:44.966 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72635.log 2024-09-15T01:12:44.967 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62612.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62612.log.gz 2024-09-15T01:12:44.967 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60368.log 2024-09-15T01:12:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72635.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72635.log.gz 2024-09-15T01:12:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49371.log 2024-09-15T01:12:44.969 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40722.log.gz 2024-09-15T01:12:44.969 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60368.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60368.log.gz 2024-09-15T01:12:44.970 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45334.log 2024-09-15T01:12:44.971 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.49371.log: -- /var/log/ceph/cluster2-client.mirror.55260.log 2024-09-15T01:12:44.972 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45334.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49371.log.gz 2024-09-15T01:12:44.972 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45334.log.gz 2024-09-15T01:12:44.972 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36742.log 2024-09-15T01:12:44.973 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64743.log 2024-09-15T01:12:44.973 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55260.log.gz 2024-09-15T01:12:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.36742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55356.log 2024-09-15T01:12:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.64743.log: -- /var/log/ceph/cluster1-client.mirror.53814.log 2024-09-15T01:12:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55356.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64743.log.gz 2024-09-15T01:12:44.976 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55356.log.gz 2024-09-15T01:12:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60105.log 2024-09-15T01:12:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50936.log 2024-09-15T01:12:44.976 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53814.log.gz 2024-09-15T01:12:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37546.log 2024-09-15T01:12:44.979 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50936.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50936.log.gz 2024-09-15T01:12:44.979 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44112.log 2024-09-15T01:12:44.979 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48702.log 2024-09-15T01:12:44.979 INFO:teuthology.orchestra.run.smithi059.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60105.log.gz 2024-09-15T01:12:44.979 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69535.log 2024-09-15T01:12:44.979 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37546.log.gz 2024-09-15T01:12:44.979 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44112.log: /var/log/ceph/cluster2-client.mirror.48702.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48702.log.gz 2024-09-15T01:12:44.980 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70036.log 2024-09-15T01:12:44.980 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69535.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69535.log.gz 2024-09-15T01:12:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51089.log 2024-09-15T01:12:44.982 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70036.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36742.log.gz 2024-09-15T01:12:44.983 INFO:teuthology.orchestra.run.smithi059.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70036.log.gz 2024-09-15T01:12:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.52174.log 2024-09-15T01:12:44.984 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51089.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51089.log.gz 2024-09-15T01:12:44.985 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64102.log 2024-09-15T01:12:44.986 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.1.52174.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44112.log.gz 2024-09-15T01:12:44.986 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45297.log 2024-09-15T01:12:44.999 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64102.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64102.log.gz 2024-09-15T01:12:44.999 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41536.log 2024-09-15T01:12:44.999 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45297.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45297.log.gz 2024-09-15T01:12:44.999 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45631.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62528.log 2024-09-15T01:12:44.999 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:44.999 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.52174.log.gz 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45688.log 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61584.log 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45688.log: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44922.log 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62528.log.gz 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45688.log.gz 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69620.log 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61584.log: /var/log/ceph/cluster1-client.mirror.44922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51250.log 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61584.log.gz 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69620.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44922.log.gz 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58614.log 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69620.log.gz 2024-09-15T01:12:45.000 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.51250.log: -- /var/log/ceph/cluster1-client.mirror.40185.log 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71829.log 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58614.log.gz 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56837.log 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71829.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71829.log.gz 92.1% 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51250.log.gz 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40185.log.gz 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54336.log 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.63661.log 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56837.log.gz 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46370.log 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54336.log.gz 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41230.log 2024-09-15T01:12:45.001 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.2.63661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50438.log 2024-09-15T01:12:45.002 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46151.log.gz 2024-09-15T01:12:45.002 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46370.log: /var/log/ceph/cluster1-client.mirror.41230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66474.log 2024-09-15T01:12:45.002 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59654.log 2024-09-15T01:12:45.002 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66474.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46370.log.gz 2024-09-15T01:12:45.002 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50438.log.gz 2024-09-15T01:12:45.002 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66474.log.gz 2024-09-15T01:12:45.002 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7%/var/log/ceph/cluster1-client.mirror.59654.log: -- replaced with /var/log/ceph/cluster1-client.mirror.41230.log.gz 2024-09-15T01:12:45.019 INFO:teuthology.orchestra.run.smithi059.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41536.log.gz 2024-09-15T01:12:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43897.log 2024-09-15T01:12:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49985.log 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45668.log 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59654.log.gz 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43897.log: /var/log/ceph/cluster2-client.mirror.49985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55240.log 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45668.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49985.log.gz 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45668.log.gz 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50121.log 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64702.log 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50121.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55240.log.gz 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69366.log 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64702.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50121.log.gz 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67464.log 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64702.log.gz 2024-09-15T01:12:45.020 INFO:teuthology.orchestra.run.smithi059.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.63661.log.gz 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69366.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42731.log 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69366.log.gz/var/log/ceph/cluster1-client.mirror.67464.log: 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43748.log 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67464.log.gz 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43897.log.gz 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66272.log 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42731.log.gz 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose/var/log/ceph/cluster2-client.mirror.43748.log: -- /var/log/ceph/cluster1-client.mirror.43811.log 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66272.log: gzip -5 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66272.log.gz 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.35912.log 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45710.log 2024-09-15T01:12:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35912.log: gzip 0.0% -5 --verbose -- /var/log/ceph/cluster2-client.admin.35825.log 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.35912.log.gz 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39967.log 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45710.log: /var/log/ceph/cluster2-client.admin.35825.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35825.log.gz 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36206.log 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.39967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60731.log 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.36206.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36206.log.gz 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40141.log 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60731.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60731.log.gz 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.43748.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.admin.72425.log 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47729.log 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43811.log.gz/var/log/ceph/cluster2-client.admin.72425.log: 2024-09-15T01:12:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46255.log 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72425.log.gz 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47589.log 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60241.log 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46255.log.gz 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47729.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40141.log.gz 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47589.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47589.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55035.log 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50165.log 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60241.log.gz 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71716.log 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50165.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55035.log.gz 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50165.log.gz 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45045.log 2024-09-15T01:12:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45549.log 2024-09-15T01:12:45.024 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45045.log: /var/log/ceph/cluster1-client.mirror.71716.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71716.log.gz 2024-09-15T01:12:45.024 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60325.log 2024-09-15T01:12:45.024 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45045.log.gz/var/log/ceph/cluster2-client.mirror.45549.log: 2024-09-15T01:12:45.024 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41720.log 2024-09-15T01:12:45.026 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60325.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60325.log.gz 2024-09-15T01:12:45.026 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43510.log 2024-09-15T01:12:45.026 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41720.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39967.log.gz 2024-09-15T01:12:45.026 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41720.log.gz 2024-09-15T01:12:45.026 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48168.log 2024-09-15T01:12:45.030 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43510.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43510.log.gz 2024-09-15T01:12:45.031 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53951.log 2024-09-15T01:12:45.031 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48168.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48168.log.gz 2024-09-15T01:12:45.032 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47136.log 2024-09-15T01:12:45.034 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53951.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53951.log.gz 2024-09-15T01:12:45.034 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66924.log 2024-09-15T01:12:45.034 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47136.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47136.log.gz 2024-09-15T01:12:45.036 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51956.log 2024-09-15T01:12:45.036 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66924.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66924.log.gz 2024-09-15T01:12:45.037 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64268.log 2024-09-15T01:12:45.037 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51956.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51956.log.gz 2024-09-15T01:12:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38679.log 2024-09-15T01:12:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64268.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64268.log.gz 2024-09-15T01:12:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47276.log 2024-09-15T01:12:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.38679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41600.log 2024-09-15T01:12:45.040 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38679.log.gz 2024-09-15T01:12:45.042 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47276.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47276.log.gz 2024-09-15T01:12:45.042 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71954.log 2024-09-15T01:12:45.042 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54674.log 2024-09-15T01:12:45.043 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41600.log.gz 2024-09-15T01:12:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71954.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71954.log.gz 2024-09-15T01:12:45.044 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51069.log 2024-09-15T01:12:45.046 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54674.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54674.log.gz 2024-09-15T01:12:45.046 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66294.log 2024-09-15T01:12:45.047 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51069.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51069.log.gz 2024-09-15T01:12:45.050 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61359.log 2024-09-15T01:12:45.050 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66294.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48320.log.gz 2024-09-15T01:12:45.050 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66294.log.gz 2024-09-15T01:12:45.050 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56540.log 2024-09-15T01:12:45.051 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62341.log 2024-09-15T01:12:45.051 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56540.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61359.log.gz 2024-09-15T01:12:45.051 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56520.log 2024-09-15T01:12:45.051 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62341.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56540.log.gz 2024-09-15T01:12:45.051 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62341.log.gz 2024-09-15T01:12:45.053 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69853.log 2024-09-15T01:12:45.053 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65718.log 2024-09-15T01:12:45.053 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69853.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56520.log.gz 2024-09-15T01:12:45.053 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62461.log 2024-09-15T01:12:45.055 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65718.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65718.log.gz 2024-09-15T01:12:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37626.log 2024-09-15T01:12:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.62461.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62461.log.gz 2024-09-15T01:12:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58958.log 2024-09-15T01:12:45.057 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65221.log 2024-09-15T01:12:45.059 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58958.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58958.log.gz 2024-09-15T01:12:45.059 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45831.log 2024-09-15T01:12:45.061 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65221.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65221.log.gz 2024-09-15T01:12:45.061 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63896.log 2024-09-15T01:12:45.063 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45831.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45831.log.gz 2024-09-15T01:12:45.063 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71760.log 2024-09-15T01:12:45.064 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63896.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63896.log.gz 2024-09-15T01:12:45.065 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59363.log 2024-09-15T01:12:45.066 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71760.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71760.log.gz 2024-09-15T01:12:45.066 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72299.log 2024-09-15T01:12:45.068 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59363.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59363.log.gz 2024-09-15T01:12:45.068 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.49647.log 2024-09-15T01:12:45.068 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72299.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72299.log.gz 2024-09-15T01:12:45.069 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44857.log 2024-09-15T01:12:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.0.49647.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.49647.log.gz 2024-09-15T01:12:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72073.log 2024-09-15T01:12:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44857.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44857.log.gz 2024-09-15T01:12:45.077 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.43934.log 2024-09-15T01:12:45.077 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.72073.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72073.log.gz 2024-09-15T01:12:45.077 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49253.log.gz 2024-09-15T01:12:45.077 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.67217.log 2024-09-15T01:12:45.083 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.0.43934.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.43934.log.gz 2024-09-15T01:12:45.084 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60434.log 2024-09-15T01:12:45.085 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67217.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67217.log.gz 2024-09-15T01:12:45.085 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61984.log 2024-09-15T01:12:45.087 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60434.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60434.log.gz 2024-09-15T01:12:45.087 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62072.log 2024-09-15T01:12:45.089 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61984.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61984.log.gz 2024-09-15T01:12:45.089 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59000.log 2024-09-15T01:12:45.090 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62072.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62072.log.gz 2024-09-15T01:12:45.091 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35492.log 2024-09-15T01:12:45.093 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59000.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59000.log.gz 2024-09-15T01:12:45.093 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63211.log 2024-09-15T01:12:45.093 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35492.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35492.log.gz 2024-09-15T01:12:45.094 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67779.log 2024-09-15T01:12:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63211.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63211.log.gz 2024-09-15T01:12:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67352.log 2024-09-15T01:12:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67779.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67779.log.gz 2024-09-15T01:12:45.098 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60217.log 2024-09-15T01:12:45.098 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67352.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67352.log.gz 2024-09-15T01:12:45.099 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61673.log 2024-09-15T01:12:45.100 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60217.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60217.log.gz 2024-09-15T01:12:45.100 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40410.log 2024-09-15T01:12:45.102 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61673.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61673.log.gz 2024-09-15T01:12:45.103 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70016.log 2024-09-15T01:12:45.104 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35332.log 2024-09-15T01:12:45.104 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70016.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70016.log.gz 2024-09-15T01:12:45.105 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59114.log 2024-09-15T01:12:45.106 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35332.log.gz 2024-09-15T01:12:45.106 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35574.log 2024-09-15T01:12:45.107 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59114.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59114.log.gz 2024-09-15T01:12:45.111 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35574.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35574.log.gz 2024-09-15T01:12:45.113 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53187.log 2024-09-15T01:12:45.114 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49470.log 2024-09-15T01:12:45.114 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37507.log 2024-09-15T01:12:45.114 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49470.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49470.log.gz 2024-09-15T01:12:45.115 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53187.log.gz 2024-09-15T01:12:45.116 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57239.log 2024-09-15T01:12:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60530.log 2024-09-15T01:12:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57239.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57239.log.gz 2024-09-15T01:12:45.117 INFO:teuthology.orchestra.run.smithi059.stderr: 53.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37507.log.gz 2024-09-15T01:12:45.119 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70581.log 2024-09-15T01:12:45.119 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60530.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60530.log.gz 2024-09-15T01:12:45.119 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70581.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70581.log.gz 2024-09-15T01:12:45.121 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66947.log 2024-09-15T01:12:45.123 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54903.log 2024-09-15T01:12:45.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43208.log 2024-09-15T01:12:45.123 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66947.log.gz 2024-09-15T01:12:45.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40209.log 2024-09-15T01:12:45.125 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43208.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54903.log.gz 2024-09-15T01:12:45.126 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43208.log.gz 2024-09-15T01:12:45.128 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46857.log 2024-09-15T01:12:45.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64660.log 2024-09-15T01:12:45.128 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51161.log 2024-09-15T01:12:45.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46857.log: /var/log/ceph/cluster2-client.mirror.64660.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40209.log.gz 2024-09-15T01:12:45.135 INFO:teuthology.orchestra.run.smithi059.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64660.log.gz 2024-09-15T01:12:45.135 INFO:teuthology.orchestra.run.smithi059.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46857.log.gz 2024-09-15T01:12:45.135 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63751.log 2024-09-15T01:12:45.135 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53252.log 2024-09-15T01:12:45.135 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51161.log.gz 2024-09-15T01:12:45.135 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68072.log 2024-09-15T01:12:45.136 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63751.log.gz 2024-09-15T01:12:45.137 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53252.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53252.log.gz 2024-09-15T01:12:45.138 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/cluster1-client.mirror.68072.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68072.log.gz 2024-09-15T01:12:45.139 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.36662.log 2024-09-15T01:12:45.142 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.2.36662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57464.log 2024-09-15T01:12:45.143 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67914.log 2024-09-15T01:12:45.143 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50912.log 2024-09-15T01:12:45.144 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67914.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67914.log.gz 2024-09-15T01:12:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57464.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57464.log.gz 2024-09-15T01:12:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55605.log 2024-09-15T01:12:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50912.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50912.log.gz 2024-09-15T01:12:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35276.log 2024-09-15T01:12:45.148 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55605.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55605.log.gz 2024-09-15T01:12:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35276.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35276.log.gz 2024-09-15T01:12:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.46608.log 2024-09-15T01:12:45.153 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.2.46608.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55443.log 2024-09-15T01:12:45.154 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-15T01:12:45.154 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55443.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55443.log.gz 2024-09-15T01:12:45.157 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69773.log 2024-09-15T01:12:45.161 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-osd.2.log: /var/log/ceph/cluster2-client.mirror.69773.log: gzip -5 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69773.log.gz 2024-09-15T01:12:45.162 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.54087.log 2024-09-15T01:12:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55650.log 2024-09-15T01:12:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54087.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54087.log.gz 2024-09-15T01:12:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43136.log 2024-09-15T01:12:45.164 INFO:teuthology.orchestra.run.smithi059.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.46608.log.gz 2024-09-15T01:12:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43854.log 2024-09-15T01:12:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43136.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43136.log.gz 2024-09-15T01:12:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51386.log 2024-09-15T01:12:45.166 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55650.log: /var/log/ceph/cluster1-client.mirror.43854.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.2.36662.log.gz 2024-09-15T01:12:45.168 INFO:teuthology.orchestra.run.smithi059.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55650.log.gz 2024-09-15T01:12:45.169 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52938.log 2024-09-15T01:12:45.169 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51386.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51386.log.gz 2024-09-15T01:12:45.170 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67442.log 2024-09-15T01:12:45.171 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52938.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52938.log.gz 2024-09-15T01:12:45.171 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70442.log 2024-09-15T01:12:45.171 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67442.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67442.log.gz 2024-09-15T01:12:45.172 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43854.log.gz 2024-09-15T01:12:45.175 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66114.log 2024-09-15T01:12:45.175 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49924.log 2024-09-15T01:12:45.175 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66114.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70442.log.gz 2024-09-15T01:12:45.175 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66114.log.gz 2024-09-15T01:12:45.175 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53274.log 2024-09-15T01:12:45.175 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.49924.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49924.log.gz 2024-09-15T01:12:45.175 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45394.log 2024-09-15T01:12:45.176 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.45394.log: 81.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45394.log.gz 2024-09-15T01:12:45.179 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64309.log 2024-09-15T01:12:45.179 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43396.log 2024-09-15T01:12:45.180 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64309.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53274.log.gz 2024-09-15T01:12:45.180 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64309.log.gz 2024-09-15T01:12:45.180 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66834.log 2024-09-15T01:12:45.181 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63624.log 2024-09-15T01:12:45.181 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43396.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43396.log.gz 2024-09-15T01:12:45.181 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71801.log 2024-09-15T01:12:45.184 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63624.log: /var/log/ceph/cluster1-client.mirror.66834.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63624.log.gz 2024-09-15T01:12:45.184 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66834.log.gz 2024-09-15T01:12:45.184 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72257.log 2024-09-15T01:12:45.185 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43934.log 2024-09-15T01:12:45.188 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71801.log: 91.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71801.log.gz 2024-09-15T01:12:45.188 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59387.log 2024-09-15T01:12:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43934.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43934.log.gz 2024-09-15T01:12:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68176.log 2024-09-15T01:12:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72257.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72257.log.gz 2024-09-15T01:12:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39047.log 2024-09-15T01:12:45.191 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59387.log.gz 2024-09-15T01:12:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68869.log 2024-09-15T01:12:45.192 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.39047.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70974.log 2024-09-15T01:12:45.194 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68869.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39047.log.gz 2024-09-15T01:12:45.194 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68869.log.gz 2024-09-15T01:12:45.194 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68176.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68176.log.gz 2024-09-15T01:12:45.195 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51932.log 2024-09-15T01:12:45.195 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70974.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70974.log.gz 2024-09-15T01:12:45.196 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70425.log 2024-09-15T01:12:45.196 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51932.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51932.log.gz 2024-09-15T01:12:45.197 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55152.log 2024-09-15T01:12:45.198 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70425.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70425.log.gz 2024-09-15T01:12:45.199 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64516.log 2024-09-15T01:12:45.199 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44776.log 2024-09-15T01:12:45.199 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64516.log: /var/log/ceph/cluster2-client.mirror.55152.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64516.log.gz 2024-09-15T01:12:45.201 INFO:teuthology.orchestra.run.smithi059.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55152.log.gz 2024-09-15T01:12:45.201 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37285.log 2024-09-15T01:12:45.203 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44776.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44776.log.gz 2024-09-15T01:12:45.203 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59046.log 2024-09-15T01:12:45.203 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60844.log 2024-09-15T01:12:45.204 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.37285.log: /var/log/ceph/cluster2-client.mirror.59046.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59225.log 2024-09-15T01:12:45.205 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60844.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59046.log.gz 2024-09-15T01:12:45.205 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48399.log.gz 2024-09-15T01:12:45.206 INFO:teuthology.orchestra.run.smithi059.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60844.log.gz 2024-09-15T01:12:45.206 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46715.log 2024-09-15T01:12:45.207 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43768.log 2024-09-15T01:12:45.207 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59225.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37285.log.gz 2024-09-15T01:12:45.207 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46715.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59225.log.gz 2024-09-15T01:12:45.209 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55059.log 2024-09-15T01:12:45.209 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46715.log.gz 2024-09-15T01:12:45.209 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43768.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49194.log 2024-09-15T01:12:45.209 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66519.log 2024-09-15T01:12:45.210 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49194.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49194.log.gz 2024-09-15T01:12:45.211 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56657.log 2024-09-15T01:12:45.212 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66519.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66519.log.gz 2024-09-15T01:12:45.212 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43619.log 2024-09-15T01:12:45.216 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56657.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56657.log.gz 2024-09-15T01:12:45.216 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43768.log.gz 2024-09-15T01:12:45.216 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55059.log.gz 2024-09-15T01:12:45.216 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66407.log 2024-09-15T01:12:45.216 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60617.log 2024-09-15T01:12:45.216 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66407.log: /var/log/ceph/cluster2-client.mirror.43619.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66407.log.gz 2024-09-15T01:12:45.216 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59838.log 2024-09-15T01:12:45.218 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60617.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54762.log 2024-09-15T01:12:45.219 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59838.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60617.log.gz 2024-09-15T01:12:45.219 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59838.log.gz 2024-09-15T01:12:45.219 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54180.log 2024-09-15T01:12:45.219 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35357.log 2024-09-15T01:12:45.219 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54180.log: /var/log/ceph/cluster2-client.mirror.54762.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54762.log.gz 2024-09-15T01:12:45.223 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52737.log 2024-09-15T01:12:45.223 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48009.log 2024-09-15T01:12:45.223 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52737.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43619.log.gz 2024-09-15T01:12:45.223 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52737.log.gz 2024-09-15T01:12:45.223 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58590.log 2024-09-15T01:12:45.223 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48009.log: gzip 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54180.log.gz 2024-09-15T01:12:45.223 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62233.log 2024-09-15T01:12:45.223 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46449.log 2024-09-15T01:12:45.223 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58590.log.gz 2024-09-15T01:12:45.223 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48009.log.gz 2024-09-15T01:12:45.224 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57014.log 2024-09-15T01:12:45.225 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46449.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46449.log.gz 2024-09-15T01:12:45.225 INFO:teuthology.orchestra.run.smithi059.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62233.log.gz 2024-09-15T01:12:45.225 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64330.log 2024-09-15T01:12:45.226 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49796.log 2024-09-15T01:12:45.226 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57014.log: /var/log/ceph/cluster2-client.mirror.64330.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57014.log.gz 2024-09-15T01:12:45.228 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64330.log.gz 2024-09-15T01:12:45.228 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39142.log 2024-09-15T01:12:45.228 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49796.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71357.log 2024-09-15T01:12:45.228 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62164.log 2024-09-15T01:12:45.229 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49796.log.gz 2024-09-15T01:12:45.229 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43533.log 2024-09-15T01:12:45.229 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71357.log: /var/log/ceph/cluster1-client.mirror.62164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44456.log 2024-09-15T01:12:45.229 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43533.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62164.log.gz 2024-09-15T01:12:45.230 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54063.log 2024-09-15T01:12:45.230 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5%/var/log/ceph/cluster2-client.mirror.44456.log: -- replaced with /var/log/ceph/cluster2-client.mirror.71357.log.gz 2024-09-15T01:12:45.230 INFO:teuthology.orchestra.run.smithi059.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39142.log.gz 2024-09-15T01:12:45.231 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44456.log.gz 2024-09-15T01:12:45.232 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47376.log 2024-09-15T01:12:45.232 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63854.log 2024-09-15T01:12:45.233 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54063.log: /var/log/ceph/cluster2-client.mirror.47376.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54063.log.gz 2024-09-15T01:12:45.233 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47376.log.gz 2024-09-15T01:12:45.233 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42490.log 2024-09-15T01:12:45.236 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63854.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43533.log.gz 2024-09-15T01:12:45.236 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63854.log.gz 2024-09-15T01:12:45.236 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69732.log 2024-09-15T01:12:45.236 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38795.log 2024-09-15T01:12:45.238 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42490.log: /var/log/ceph/cluster2-client.mirror.69732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40831.log 2024-09-15T01:12:45.238 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.38795.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69732.log.gz 2024-09-15T01:12:45.238 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39910.log 2024-09-15T01:12:45.238 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.40831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59496.log 2024-09-15T01:12:45.238 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51683.log 2024-09-15T01:12:45.239 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59496.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59496.log.gz 2024-09-15T01:12:45.240 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52577.log 2024-09-15T01:12:45.240 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.40831.log.gz 2024-09-15T01:12:45.242 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51683.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51683.log.gz 2024-09-15T01:12:45.242 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38795.log.gz 2024-09-15T01:12:45.242 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.55687.log 2024-09-15T01:12:45.242 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38540.log 2024-09-15T01:12:45.243 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.1.55687.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52577.log.gz 2024-09-15T01:12:45.245 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47749.log 2024-09-15T01:12:45.263 INFO:teuthology.orchestra.run.smithi059.stderr: 92.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51410.log 2024-09-15T01:12:45.263 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47749.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47749.log.gz 2024-09-15T01:12:45.263 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.55687.log.gz 2024-09-15T01:12:45.263 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.38540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41384.log 2024-09-15T01:12:45.266 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51410.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51410.log.gz 2024-09-15T01:12:45.266 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39252.log 2024-09-15T01:12:45.266 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43232.log 2024-09-15T01:12:45.266 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47766.log 2024-09-15T01:12:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43232.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43232.log.gz 2024-09-15T01:12:45.268 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63459.log 2024-09-15T01:12:45.268 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38540.log.gz 2024-09-15T01:12:45.268 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47766.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41384.log.gz 2024-09-15T01:12:45.269 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47766.log.gz 2024-09-15T01:12:45.270 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44877.log 2024-09-15T01:12:45.270 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63459.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63459.log.gz 2024-09-15T01:12:45.271 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71213.log 2024-09-15T01:12:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44877.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39252.log.gz 2024-09-15T01:12:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45814.log 2024-09-15T01:12:45.276 INFO:teuthology.orchestra.run.smithi059.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44877.log.gz 2024-09-15T01:12:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42872.log 2024-09-15T01:12:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45814.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45814.log.gz 2024-09-15T01:12:45.277 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71213.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71213.log.gz 2024-09-15T01:12:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37473.log 2024-09-15T01:12:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42872.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65180.log 2024-09-15T01:12:45.280 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64082.log 2024-09-15T01:12:45.280 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65180.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65180.log.gz 2024-09-15T01:12:45.280 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47887.log 2024-09-15T01:12:45.283 INFO:teuthology.orchestra.run.smithi059.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42872.log.gz 2024-09-15T01:12:45.284 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63542.log 2024-09-15T01:12:45.284 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47887.log: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37473.log.gz 2024-09-15T01:12:45.291 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64082.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64082.log.gz 2024-09-15T01:12:45.293 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48618.log 2024-09-15T01:12:45.293 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63542.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.admin.35357.log.gz 2024-09-15T01:12:45.293 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63542.log.gz 2024-09-15T01:12:45.294 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58164.log 2024-09-15T01:12:45.294 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48618.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48618.log.gz 2024-09-15T01:12:45.295 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62880.log 2024-09-15T01:12:45.296 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58164.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58164.log.gz 2024-09-15T01:12:45.296 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.36742.log 2024-09-15T01:12:45.301 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62880.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62880.log.gz 2024-09-15T01:12:45.301 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65449.log 2024-09-15T01:12:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.0.36742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69794.log 2024-09-15T01:12:45.312 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65449.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65449.log.gz 2024-09-15T01:12:45.314 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58686.log 2024-09-15T01:12:45.314 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69794.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69794.log.gz 2024-09-15T01:12:45.314 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64433.log 2024-09-15T01:12:45.315 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58686.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58686.log.gz 2024-09-15T01:12:45.316 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58707.log 2024-09-15T01:12:45.316 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64433.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64433.log.gz 2024-09-15T01:12:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66677.log 2024-09-15T01:12:45.322 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58707.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58707.log.gz 2024-09-15T01:12:45.323 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71781.log 2024-09-15T01:12:45.323 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66677.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66677.log.gz 2024-09-15T01:12:45.324 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35660.log 2024-09-15T01:12:45.324 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71781.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71781.log.gz 2024-09-15T01:12:45.328 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66609.log 2024-09-15T01:12:45.332 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35660.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35660.log.gz 2024-09-15T01:12:45.332 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56926.log 2024-09-15T01:12:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66609.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66609.log.gz 2024-09-15T01:12:45.333 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47729.log.gz 2024-09-15T01:12:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53163.log 2024-09-15T01:12:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40765.log 2024-09-15T01:12:45.335 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53163.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53163.log.gz 2024-09-15T01:12:45.335 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43338.log 2024-09-15T01:12:45.335 INFO:teuthology.orchestra.run.smithi059.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56926.log.gz 2024-09-15T01:12:45.336 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69886.log 2024-09-15T01:12:45.336 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41888.log 2024-09-15T01:12:45.336 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69886.log: /var/log/ceph/cluster2-client.mirror.43338.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69886.log.gz 2024-09-15T01:12:45.340 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36122.log 2024-09-15T01:12:45.340 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41888.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43338.log.gz 2024-09-15T01:12:45.340 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57536.log 2024-09-15T01:12:45.340 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55172.log 2024-09-15T01:12:45.340 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57536.log: /var/log/ceph/cluster2-client.admin.36122.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36122.log.gz 2024-09-15T01:12:45.341 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57536.log.gz 2024-09-15T01:12:45.341 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45993.log 2024-09-15T01:12:45.341 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46698.log 2024-09-15T01:12:45.341 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45993.log: /var/log/ceph/cluster1-client.mirror.55172.log: 86.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45993.log.gz 2024-09-15T01:12:45.341 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45128.log 2024-09-15T01:12:45.342 INFO:teuthology.orchestra.run.smithi059.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55172.log.gz 2024-09-15T01:12:45.342 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51230.log 2024-09-15T01:12:45.344 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45128.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45128.log.gz 2024-09-15T01:12:45.344 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71299.log 2024-09-15T01:12:45.346 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51230.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51230.log.gz 2024-09-15T01:12:45.347 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51567.log 2024-09-15T01:12:45.348 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71299.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71299.log.gz 2024-09-15T01:12:45.348 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50591.log 2024-09-15T01:12:45.348 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51567.log: 73.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41888.log.gz 2024-09-15T01:12:45.349 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51567.log.gz 2024-09-15T01:12:45.350 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.70162.log 2024-09-15T01:12:45.351 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50591.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50591.log.gz 2024-09-15T01:12:45.351 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44026.log 2024-09-15T01:12:45.355 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.2.70162.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.70162.log.gz 2024-09-15T01:12:45.356 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64474.log 2024-09-15T01:12:45.357 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44026.log: /var/log/ceph/cluster2-client.mirror.46698.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46698.log.gz 2024-09-15T01:12:45.358 INFO:teuthology.orchestra.run.smithi059.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37626.log.gz 2024-09-15T01:12:45.360 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40765.log.gz 2024-09-15T01:12:45.360 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37529.log 2024-09-15T01:12:45.362 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44026.log.gz 2024-09-15T01:12:45.363 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52287.log 2024-09-15T01:12:45.364 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68235.log 2024-09-15T01:12:45.365 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52287.log: /var/log/ceph/cluster2-client.mirror.64474.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52287.log.gz 2024-09-15T01:12:45.365 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66699.log 2024-09-15T01:12:45.366 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68235.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64474.log.gz 2024-09-15T01:12:45.368 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66429.log 2024-09-15T01:12:45.368 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66699.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66699.log.gz 2024-09-15T01:12:45.368 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54039.log 2024-09-15T01:12:45.368 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66429.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66429.log.gz 2024-09-15T01:12:45.370 INFO:teuthology.orchestra.run.smithi059.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37529.log.gz 2024-09-15T01:12:45.381 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45954.log 2024-09-15T01:12:45.385 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54039.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54039.log.gz 2024-09-15T01:12:45.385 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65553.log 2024-09-15T01:12:45.398 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65780.log 2024-09-15T01:12:45.400 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65553.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65553.log.gz 2024-09-15T01:12:45.426 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39074.log 2024-09-15T01:12:45.427 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65780.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65780.log.gz 2024-09-15T01:12:45.433 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59766.log 2024-09-15T01:12:45.442 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.39074.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63232.log 2024-09-15T01:12:45.446 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59766.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39074.log.gz 2024-09-15T01:12:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49986.log 2024-09-15T01:12:45.454 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59766.log.gz 2024-09-15T01:12:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63232.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63232.log.gz 2024-09-15T01:12:45.462 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63958.log 2024-09-15T01:12:45.471 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.49986.log: 95.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61492.log 2024-09-15T01:12:45.477 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63958.log: -- replaced with /var/log/ceph/cluster1-client.mirror.49986.log.gz 2024-09-15T01:12:45.484 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63958.log.gz 2024-09-15T01:12:45.495 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69174.log 2024-09-15T01:12:45.497 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61492.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61492.log.gz 2024-09-15T01:12:45.503 INFO:teuthology.orchestra.run.smithi059.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.36742.log.gz 2024-09-15T01:12:45.508 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63397.log 2024-09-15T01:12:45.510 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69174.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69174.log.gz 2024-09-15T01:12:45.517 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54200.log 2024-09-15T01:12:45.527 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63397.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61222.log 2024-09-15T01:12:45.531 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54200.log: 92.3% 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63397.log.gz 2024-09-15T01:12:45.541 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54200.log.gz 2024-09-15T01:12:45.544 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40388.log 2024-09-15T01:12:45.553 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61222.log: 90.3% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45087.log.gz 2024-09-15T01:12:45.553 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.61222.log.gz 2024-09-15T01:12:45.554 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62140.log 2024-09-15T01:12:45.555 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40388.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63500.log 2024-09-15T01:12:45.556 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.62140.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62140.log.gz 2024-09-15T01:12:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54513.log 2024-09-15T01:12:45.560 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63500.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40388.log.gz 2024-09-15T01:12:45.560 INFO:teuthology.orchestra.run.smithi059.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63500.log.gz 2024-09-15T01:12:45.561 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71116.log 2024-09-15T01:12:45.562 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54513.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54513.log.gz 2024-09-15T01:12:45.562 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72097.log 2024-09-15T01:12:45.574 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44736.log 2024-09-15T01:12:45.574 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72097.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72097.log.gz 2024-09-15T01:12:45.591 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42979.log 2024-09-15T01:12:45.592 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44736.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44736.log.gz 2024-09-15T01:12:45.605 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48108.log 2024-09-15T01:12:45.611 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71340.log 2024-09-15T01:12:45.613 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48108.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48108.log.gz 2024-09-15T01:12:45.619 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41804.log 2024-09-15T01:12:45.619 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71340.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71340.log.gz 2024-09-15T01:12:45.628 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49736.log 2024-09-15T01:12:45.647 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43316.log 2024-09-15T01:12:45.647 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49736.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49736.log.gz 2024-09-15T01:12:45.648 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41804.log.gz 2024-09-15T01:12:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47707.log 2024-09-15T01:12:45.661 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43316.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43316.log.gz 2024-09-15T01:12:45.675 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36822.log 2024-09-15T01:12:45.675 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47707.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47707.log.gz 2024-09-15T01:12:45.696 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41980.log 2024-09-15T01:12:45.701 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.36822.log: 74.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36822.log.gz 2024-09-15T01:12:45.708 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41342.log 2024-09-15T01:12:45.708 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.41980.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.41980.log.gz 2024-09-15T01:12:45.726 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-15T01:12:45.730 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41342.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68235.log.gz 2024-09-15T01:12:45.744 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48457.log 2024-09-15T01:12:45.750 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2.audit.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69853.log.gz 2024-09-15T01:12:45.751 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47259.log 2024-09-15T01:12:45.755 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48457.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48457.log.gz 2024-09-15T01:12:45.755 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42403.log 2024-09-15T01:12:45.755 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47259.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47259.log.gz 2024-09-15T01:12:45.755 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71278.log 2024-09-15T01:12:45.756 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42403.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41342.log.gz 2024-09-15T01:12:45.766 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42403.log.gz 2024-09-15T01:12:45.766 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44536.log 2024-09-15T01:12:45.766 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71278.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71278.log.gz 2024-09-15T01:12:45.766 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39516.log 2024-09-15T01:12:45.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44536.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44536.log.gz 2024-09-15T01:12:45.769 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52599.log 2024-09-15T01:12:45.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65346.log 2024-09-15T01:12:45.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52599.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52599.log.gz 2024-09-15T01:12:45.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43791.log 2024-09-15T01:12:45.777 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45549.log.gz 2024-09-15T01:12:45.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65346.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65346.log.gz 2024-09-15T01:12:45.781 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47887.log.gz 2024-09-15T01:12:45.784 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45111.log 2024-09-15T01:12:45.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57287.log 2024-09-15T01:12:45.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43791.log: /var/log/ceph/cluster2-client.mirror.45111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49861.log 2024-09-15T01:12:45.786 INFO:teuthology.orchestra.run.smithi059.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45111.log.gz 2024-09-15T01:12:45.789 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57287.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57287.log.gz 2024-09-15T01:12:45.789 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48804.log 2024-09-15T01:12:45.789 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41027.log 2024-09-15T01:12:45.792 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48804.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43791.log.gz 2024-09-15T01:12:45.792 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47216.log 2024-09-15T01:12:45.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41027.log: gzip/var/log/ceph/cluster2-client.mirror.49861.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62694.log 2024-09-15T01:12:45.793 INFO:teuthology.orchestra.run.smithi059.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49861.log.gz 2024-09-15T01:12:45.793 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40681.log 2024-09-15T01:12:45.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47216.log: /var/log/ceph/cluster2-client.mirror.62694.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47216.log.gz 2024-09-15T01:12:45.807 INFO:teuthology.orchestra.run.smithi059.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62694.log.gz 2024-09-15T01:12:45.807 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62591.log 2024-09-15T01:12:45.814 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40681.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61512.log 2024-09-15T01:12:45.815 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40681.log.gz 2024-09-15T01:12:45.815 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62591.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62591.log.gz 2024-09-15T01:12:45.815 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41580.log 2024-09-15T01:12:45.815 INFO:teuthology.orchestra.run.smithi059.stderr: 92.6% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-15T01:12:45.815 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61512.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61512.log.gz 2024-09-15T01:12:45.816 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63813.log 2024-09-15T01:12:45.817 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65934.log 2024-09-15T01:12:45.824 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63813.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41580.log.gz 2024-09-15T01:12:45.825 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63813.log.gz 2024-09-15T01:12:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45710.log.gz 2024-09-15T01:12:45.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.65934.log: 87.0% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53340.log 2024-09-15T01:12:45.830 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65934.log.gz 2024-09-15T01:12:45.830 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52713.log 2024-09-15T01:12:45.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70298.log 2024-09-15T01:12:45.830 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53340.log.gz 2024-09-15T01:12:45.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52713.log: /var/log/ceph/cluster2-client.mirror.70298.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70298.log.gz 2024-09-15T01:12:45.832 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52713.log.gz 2024-09-15T01:12:45.833 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65656.log 2024-09-15T01:12:45.834 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46996.log 2024-09-15T01:12:45.834 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60554.log 2024-09-15T01:12:45.834 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46996.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65656.log.gz 2024-09-15T01:12:45.835 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60554.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60554.log.gz 2024-09-15T01:12:45.839 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39429.log 2024-09-15T01:12:45.842 INFO:teuthology.orchestra.run.smithi059.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41027.log.gz 2024-09-15T01:12:45.844 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.39429.log: --verbose -- /var/log/ceph/cluster1-client.mirror.37268.log 2024-09-15T01:12:45.844 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71854.log 2024-09-15T01:12:45.846 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.37268.log: /var/log/ceph/cluster1-client.mirror.71854.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71854.log.gz 2024-09-15T01:12:45.846 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39388.log 2024-09-15T01:12:45.846 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32548.log 2024-09-15T01:12:45.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44132.log 2024-09-15T01:12:45.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.32548.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32548.log.gz 2024-09-15T01:12:45.848 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53724.log 2024-09-15T01:12:45.854 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44132.log: 54.2% 87.6% 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44132.log.gz 2024-09-15T01:12:45.855 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36374.log 2024-09-15T01:12:45.855 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42490.log.gz 2024-09-15T01:12:45.855 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.37268.log.gz 2024-09-15T01:12:45.855 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.39388.log.gz 2024-09-15T01:12:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53724.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53724.log.gz 2024-09-15T01:12:45.858 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.49986.log 2024-09-15T01:12:45.858 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.36374.log: -- /var/log/ceph/cluster1-client.mirror.67937.log 2024-09-15T01:12:45.858 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36374.log.gz 2024-09-15T01:12:45.858 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.2.49986.log: /var/log/ceph/cluster2-client.mirror.50300.log 2024-09-15T01:12:45.859 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27537.log 2024-09-15T01:12:45.860 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50300.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50300.log.gz 2024-09-15T01:12:45.860 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62715.log 2024-09-15T01:12:45.861 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.tmp-client.admin.27537.log: /var/log/ceph/cluster1-client.mirror.67937.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27537.log.gz 2024-09-15T01:12:45.861 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67937.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63937.log 2024-09-15T01:12:45.861 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:45.861 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70516.log 2024-09-15T01:12:45.862 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62715.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62715.log.gz 2024-09-15T01:12:45.862 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67892.log 2024-09-15T01:12:45.863 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70516.log: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70516.log.gz 2024-09-15T01:12:45.864 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56431.log 2024-09-15T01:12:45.864 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67892.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67892.log.gz 2024-09-15T01:12:45.865 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39782.log 2024-09-15T01:12:45.867 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56431.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56431.log.gz 2024-09-15T01:12:45.867 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38840.log 2024-09-15T01:12:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43444.log 2024-09-15T01:12:45.870 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.38840.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52665.log 2024-09-15T01:12:45.871 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63937.log: /var/log/ceph/cluster1-client.mirror.43444.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43444.log.gz 2024-09-15T01:12:45.871 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63937.log.gz 2024-09-15T01:12:45.871 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56745.log 2024-09-15T01:12:45.872 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52665.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52665.log.gz 2024-09-15T01:12:45.872 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40702.log 2024-09-15T01:12:45.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56745.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56745.log.gz 2024-09-15T01:12:45.875 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39803.log 2024-09-15T01:12:45.875 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39782.log.gz 2024-09-15T01:12:45.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40541.log 2024-09-15T01:12:45.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62116.log 2024-09-15T01:12:45.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32611.log 2024-09-15T01:12:45.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.62116.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62116.log.gz 2024-09-15T01:12:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71739.log 2024-09-15T01:12:45.879 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.32611.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32611.log.gz 2024-09-15T01:12:45.881 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40257.log 2024-09-15T01:12:45.881 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71739.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71739.log.gz 2024-09-15T01:12:45.881 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36500.log 2024-09-15T01:12:45.883 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47078.log 2024-09-15T01:12:45.883 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.36500.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36500.log.gz 2024-09-15T01:12:45.883 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63087.log 2024-09-15T01:12:45.886 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47078.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40257.log.gz 2024-09-15T01:12:45.891 INFO:teuthology.orchestra.run.smithi059.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40541.log.gz 2024-09-15T01:12:45.893 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40702.log.gz 2024-09-15T01:12:45.893 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45006.log 2024-09-15T01:12:45.894 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63087.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63087.log.gz 2024-09-15T01:12:45.896 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54384.log 2024-09-15T01:12:45.913 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45851.log 2024-09-15T01:12:45.913 INFO:teuthology.orchestra.run.smithi059.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.49986.log.gz 2024-09-15T01:12:45.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54384.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54384.log.gz 2024-09-15T01:12:45.916 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50615.log 2024-09-15T01:12:45.917 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45851.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45851.log.gz 2024-09-15T01:12:45.917 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70342.log 2024-09-15T01:12:45.917 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50615.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50615.log.gz 2024-09-15T01:12:45.919 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60980.log 2024-09-15T01:12:45.919 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70342.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70342.log.gz 2024-09-15T01:12:45.920 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.41888.log 2024-09-15T01:12:45.921 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60980.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60980.log.gz 2024-09-15T01:12:45.921 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59291.log 2024-09-15T01:12:45.922 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.1.41888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37200.log 2024-09-15T01:12:45.923 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59291.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59291.log.gz 2024-09-15T01:12:45.923 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39186.log 2024-09-15T01:12:45.924 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.37200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67149.log 2024-09-15T01:12:45.927 INFO:teuthology.orchestra.run.smithi059.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.41888.log.gz 2024-09-15T01:12:45.927 INFO:teuthology.orchestra.run.smithi059.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37200.log.gz 2024-09-15T01:12:45.927 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47116.log 2024-09-15T01:12:45.932 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39429.log.gz 2024-09-15T01:12:45.932 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67149.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39186.log.gz 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67149.log.gz 2024-09-15T01:12:45.932 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:45.932 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37755.log 2024-09-15T01:12:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47116.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39803.log.gz 2024-09-15T01:12:45.935 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47116.log.gz 2024-09-15T01:12:45.935 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.36822.log 2024-09-15T01:12:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71319.log 2024-09-15T01:12:45.946 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.2.36822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66992.log 2024-09-15T01:12:45.953 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56474.log 2024-09-15T01:12:45.953 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66992.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66992.log.gz 2024-09-15T01:12:45.955 INFO:teuthology.orchestra.run.smithi059.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.36822.log.gz 2024-09-15T01:12:45.957 INFO:teuthology.orchestra.run.smithi059.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71319.log.gz 2024-09-15T01:12:45.960 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57693.log 2024-09-15T01:12:45.961 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56474.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56474.log.gz 2024-09-15T01:12:45.961 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.71516.log 2024-09-15T01:12:45.964 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38840.log.gz 2024-09-15T01:12:45.966 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57693.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57693.log.gz 2024-09-15T01:12:45.966 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32369.log 2024-09-15T01:12:45.971 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.0.71516.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.71516.log.gz 2024-09-15T01:12:45.971 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57110.log 2024-09-15T01:12:45.975 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.32369.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32369.log.gz 2024-09-15T01:12:45.979 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48989.log 2024-09-15T01:12:45.983 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57110.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57110.log.gz 2024-09-15T01:12:45.983 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54585.log 2024-09-15T01:12:46.001 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44218.log 2024-09-15T01:12:46.002 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54585.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54585.log.gz 2024-09-15T01:12:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69711.log 2024-09-15T01:12:46.017 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44218.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44218.log.gz 2024-09-15T01:12:46.028 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63356.log 2024-09-15T01:12:46.029 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69711.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69711.log.gz 2024-09-15T01:12:46.039 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40446.log 2024-09-15T01:12:46.040 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63356.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63356.log.gz 2024-09-15T01:12:46.048 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39662.log 2024-09-15T01:12:46.063 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.40446.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40446.log.gz 2024-09-15T01:12:46.066 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55912.log 2024-09-15T01:12:46.069 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39662.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39662.log.gz 2024-09-15T01:12:46.075 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43376.log 2024-09-15T01:12:46.088 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55912.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65056.log 2024-09-15T01:12:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43376.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55912.log.gz 2024-09-15T01:12:46.089 INFO:teuthology.orchestra.run.smithi059.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43376.log.gz 2024-09-15T01:12:46.097 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38637.log 2024-09-15T01:12:46.099 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65056.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65056.log.gz 2024-09-15T01:12:46.107 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49413.log 2024-09-15T01:12:46.119 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.38637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67374.log 2024-09-15T01:12:46.131 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58051.log 2024-09-15T01:12:46.131 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67374.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67374.log.gz 2024-09-15T01:12:46.138 INFO:teuthology.orchestra.run.smithi059.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38637.log.gz 2024-09-15T01:12:46.144 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43662.log 2024-09-15T01:12:46.144 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58051.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58051.log.gz 2024-09-15T01:12:46.153 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44947.log 2024-09-15T01:12:46.163 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43662.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68523.log 2024-09-15T01:12:46.168 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44947.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44947.log.gz 2024-09-15T01:12:46.172 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43662.log.gz 2024-09-15T01:12:46.177 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46916.log 2024-09-15T01:12:46.178 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68523.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68523.log.gz 2024-09-15T01:12:46.191 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50711.log 2024-09-15T01:12:46.201 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47056.log 2024-09-15T01:12:46.203 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50711.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50711.log.gz 2024-09-15T01:12:46.216 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59859.log 2024-09-15T01:12:46.216 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47056.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47056.log.gz 2024-09-15T01:12:46.230 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68141.log 2024-09-15T01:12:46.231 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59859.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59859.log.gz 2024-09-15T01:12:46.245 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71516.log 2024-09-15T01:12:46.245 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68141.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68141.log.gz 2024-09-15T01:12:46.253 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45954.log.gz 2024-09-15T01:12:46.262 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70094.log 2024-09-15T01:12:46.262 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71516.log: 79.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71516.log.gz 2024-09-15T01:12:46.268 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48804.log.gz 2024-09-15T01:12:46.274 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51748.log 2024-09-15T01:12:46.288 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70094.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70094.log.gz 2024-09-15T01:12:46.302 INFO:teuthology.orchestra.run.smithi059.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71116.log.gz 2024-09-15T01:12:46.302 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65912.log 2024-09-15T01:12:46.304 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51748.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51748.log.gz 2024-09-15T01:12:46.305 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65697.log 2024-09-15T01:12:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.65912.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65912.log.gz 2024-09-15T01:12:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63418.log 2024-09-15T01:12:46.308 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65697.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65697.log.gz 2024-09-15T01:12:46.309 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62901.log 2024-09-15T01:12:46.309 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63418.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63418.log.gz 2024-09-15T01:12:46.310 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.68385.log 2024-09-15T01:12:46.311 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62901.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62901.log.gz 2024-09-15T01:12:46.312 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66587.log 2024-09-15T01:12:46.324 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.2.68385.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.68385.log.gz 2024-09-15T01:12:46.324 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55421.log 2024-09-15T01:12:46.324 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66587.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66587.log.gz 2024-09-15T01:12:46.325 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65118.log 2024-09-15T01:12:46.325 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55421.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55421.log.gz 2024-09-15T01:12:46.325 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50390.log 2024-09-15T01:12:46.325 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65118.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65118.log.gz 2024-09-15T01:12:46.325 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60956.log 2024-09-15T01:12:46.325 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50390.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50390.log.gz 2024-09-15T01:12:46.325 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68889.log 2024-09-15T01:12:46.326 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60956.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60956.log.gz 2024-09-15T01:12:46.326 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64557.log 2024-09-15T01:12:46.328 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68889.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68889.log.gz 2024-09-15T01:12:46.328 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41299.log 2024-09-15T01:12:46.329 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64557.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64557.log.gz 2024-09-15T01:12:46.330 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47179.log 2024-09-15T01:12:46.337 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59814.log 2024-09-15T01:12:46.339 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47179.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47179.log.gz 2024-09-15T01:12:46.339 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44046.log 2024-09-15T01:12:46.344 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59814.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59814.log.gz 2024-09-15T01:12:46.344 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59634.log 2024-09-15T01:12:46.354 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65868.log 2024-09-15T01:12:46.354 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59634.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59634.log.gz 2024-09-15T01:12:46.365 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46996.log.gz 2024-09-15T01:12:46.366 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44046.log.gz 2024-09-15T01:12:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69386.log 2024-09-15T01:12:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.65868.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65868.log.gz 2024-09-15T01:12:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65847.log 2024-09-15T01:12:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58815.log 2024-09-15T01:12:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.65847.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65847.log.gz 2024-09-15T01:12:46.371 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69407.log 2024-09-15T01:12:46.372 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58815.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58815.log.gz 2024-09-15T01:12:46.373 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49273.log 2024-09-15T01:12:46.373 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69407.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69407.log.gz 2024-09-15T01:12:46.374 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57761.log 2024-09-15T01:12:46.374 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49273.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49273.log.gz 2024-09-15T01:12:46.376 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36038.log 2024-09-15T01:12:46.376 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57761.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57761.log.gz 2024-09-15T01:12:46.378 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70116.log 2024-09-15T01:12:46.378 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.36038.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36038.log.gz 2024-09-15T01:12:46.378 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41624.log 2024-09-15T01:12:46.380 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70116.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47078.log.gz 2024-09-15T01:12:46.380 INFO:teuthology.orchestra.run.smithi059.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69386.log.gz 2024-09-15T01:12:46.380 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51479.log 2024-09-15T01:12:46.380 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64061.log 2024-09-15T01:12:46.381 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41624.log.gz 2024-09-15T01:12:46.383 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51479.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51479.log.gz 2024-09-15T01:12:46.383 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40561.log 2024-09-15T01:12:46.384 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48437.log 2024-09-15T01:12:46.384 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64061.log.gz 2024-09-15T01:12:46.388 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54135.log 2024-09-15T01:12:46.389 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48437.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48437.log.gz 2024-09-15T01:12:46.390 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44756.log 2024-09-15T01:12:46.390 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40561.log.gz 2024-09-15T01:12:46.391 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54135.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54135.log.gz 2024-09-15T01:12:46.391 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71094.log 2024-09-15T01:12:46.392 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44756.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44756.log.gz 2024-09-15T01:12:46.394 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68327.log 2024-09-15T01:12:46.394 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49758.log 2024-09-15T01:12:46.394 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71094.log.gz 2024-09-15T01:12:46.394 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58188.log 2024-09-15T01:12:46.394 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49758.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.68327.log.gz 2024-09-15T01:12:46.403 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57647.log 2024-09-15T01:12:46.410 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58188.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58188.log.gz 2024-09-15T01:12:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54945.log 2024-09-15T01:12:46.423 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57647.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57647.log.gz 2024-09-15T01:12:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49529.log 2024-09-15T01:12:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54945.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54945.log.gz 2024-09-15T01:12:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55194.log 2024-09-15T01:12:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49529.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49529.log.gz 2024-09-15T01:12:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32236.log 2024-09-15T01:12:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55194.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55194.log.gz 2024-09-15T01:12:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61695.log 2024-09-15T01:12:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.32236.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32236.log.gz 2024-09-15T01:12:46.463 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43576.log 2024-09-15T01:12:46.474 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61695.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61695.log.gz 2024-09-15T01:12:46.474 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36542.log 2024-09-15T01:12:46.474 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50213.log 2024-09-15T01:12:46.474 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.36542.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36542.log.gz 2024-09-15T01:12:46.479 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43576.log.gz 2024-09-15T01:12:46.482 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46292.log 2024-09-15T01:12:46.487 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50213.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50213.log.gz 2024-09-15T01:12:46.487 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55629.log 2024-09-15T01:12:46.487 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46292.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46292.log.gz 2024-09-15T01:12:46.492 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42979.log.gz 2024-09-15T01:12:46.492 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39210.log 2024-09-15T01:12:46.496 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64454.log 87.6% 2024-09-15T01:12:46.496 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55629.log.gz 2024-09-15T01:12:46.497 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37166.log 2024-09-15T01:12:46.497 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64454.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64454.log.gz 2024-09-15T01:12:46.498 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55936.log 2024-09-15T01:12:46.500 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.37166.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47828.log 2024-09-15T01:12:46.500 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39210.log.gz 2024-09-15T01:12:46.500 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55936.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55936.log.gz 2024-09-15T01:12:46.500 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36164.log 2024-09-15T01:12:46.501 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47828.log: gzip 80.5% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36702.log 2024-09-15T01:12:46.501 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47828.log.gz 2024-09-15T01:12:46.501 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53661.log 2024-09-15T01:12:46.502 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.36702.log: /var/log/ceph/cluster1-client.mirror.53661.log: 66.7% -- replaced with /var/log/ceph/cluster1-client.admin.37166.log.gz 2024-09-15T01:12:46.503 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53661.log.gz 2024-09-15T01:12:46.504 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.36164.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36164.log.gz 2024-09-15T01:12:46.504 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67712.log 2024-09-15T01:12:46.504 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61632.log 2024-09-15T01:12:46.506 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67712.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67712.log.gz 2024-09-15T01:12:46.506 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61424.log 2024-09-15T01:12:46.506 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61632.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61632.log.gz 2024-09-15T01:12:46.507 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55533.log 2024-09-15T01:12:46.514 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61424.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61424.log.gz 2024-09-15T01:12:46.514 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46353.log 91.8% 2024-09-15T01:12:46.514 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36702.log.gz 2024-09-15T01:12:46.514 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55533.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55533.log.gz 2024-09-15T01:12:46.514 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51770.log 2024-09-15T01:12:46.514 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46353.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46353.log.gz 2024-09-15T01:12:46.514 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63834.log 2024-09-15T01:12:46.514 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53115.log 2024-09-15T01:12:46.514 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63834.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51770.log.gz 2024-09-15T01:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52263.log 2024-09-15T01:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53115.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53115.log.gz 2024-09-15T01:12:46.517 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42446.log 2024-09-15T01:12:46.517 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52263.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52263.log.gz 2024-09-15T01:12:46.518 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44284.log 2024-09-15T01:12:46.518 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63834.log.gz 2024-09-15T01:12:46.518 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40858.log 2024-09-15T01:12:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42446.log: /var/log/ceph/cluster2-client.mirror.44284.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42446.log.gz 2024-09-15T01:12:46.524 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65325.log 2024-09-15T01:12:46.525 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.40858.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51023.log 2024-09-15T01:12:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65325.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65325.log.gz 2024-09-15T01:12:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45507.log 2024-09-15T01:12:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51023.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.40858.log.gz 2024-09-15T01:12:46.531 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51023.log.gz 2024-09-15T01:12:46.531 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54360.log 2024-09-15T01:12:46.532 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45507.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45507.log.gz 2024-09-15T01:12:46.532 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60824.log 2024-09-15T01:12:46.533 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54360.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54360.log.gz 2024-09-15T01:12:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65387.log 2024-09-15T01:12:46.536 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60824.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60824.log.gz 2024-09-15T01:12:46.537 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41741.log 2024-09-15T01:12:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65387.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65387.log.gz 2024-09-15T01:12:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70319.log 2024-09-15T01:12:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71234.log 2024-09-15T01:12:46.541 INFO:teuthology.orchestra.run.smithi059.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44284.log.gz 2024-09-15T01:12:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70319.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70319.log.gz 2024-09-15T01:12:46.543 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51659.log 2024-09-15T01:12:46.543 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71234.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71234.log.gz 2024-09-15T01:12:46.544 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69688.log 2024-09-15T01:12:46.544 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51659.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51659.log.gz 2024-09-15T01:12:46.544 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.39251.log 2024-09-15T01:12:46.545 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56089.log 2024-09-15T01:12:46.545 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.2.39251.log: /var/log/ceph/cluster1-client.mirror.69688.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69688.log.gz 2024-09-15T01:12:46.548 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56089.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56089.log.gz 2024-09-15T01:12:46.548 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68783.log 2024-09-15T01:12:46.548 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46491.log 2024-09-15T01:12:46.549 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68783.log: gzip -5 --verbose 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68783.log.gz 2024-09-15T01:12:46.552 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46491.log: -- /var/log/ceph/cluster1-client.mirror.68479.log 2024-09-15T01:12:46.552 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-15T01:12:46.553 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68479.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68479.log.gz 2024-09-15T01:12:46.553 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46412.log 2024-09-15T01:12:46.557 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.36622.log 2024-09-15T01:12:46.566 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49211.log 2024-09-15T01:12:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.1.36622.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41299.log.gz 2024-09-15T01:12:46.572 INFO:teuthology.orchestra.run.smithi059.stderr: 72.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.36622.log.gz 2024-09-15T01:12:46.574 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64784.log 2024-09-15T01:12:46.586 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49211.log: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42779.log 2024-09-15T01:12:46.586 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64784.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49211.log.gz 2024-09-15T01:12:46.599 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56564.log 2024-09-15T01:12:46.600 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42779.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42779.log.gz 2024-09-15T01:12:46.601 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64784.log.gz 2024-09-15T01:12:46.611 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.39251.log.gz 2024-09-15T01:12:46.626 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58010.log 2024-09-15T01:12:46.627 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56564.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56564.log.gz 2024-09-15T01:12:46.647 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49838.log 2024-09-15T01:12:46.647 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58010.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58010.log.gz 2024-09-15T01:12:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59516.log 2024-09-15T01:12:46.681 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61741.log 2024-09-15T01:12:46.681 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59516.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59516.log.gz 2024-09-15T01:12:46.694 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60064.log 2024-09-15T01:12:46.695 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61741.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61741.log.gz 2024-09-15T01:12:46.702 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44304.log 2024-09-15T01:12:46.716 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60064.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60064.log.gz 2024-09-15T01:12:46.717 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39686.log 2024-09-15T01:12:46.736 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44304.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44304.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55687.log 2024-09-15T01:12:46.736 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39686.log: 2024-09-15T01:12:46.737 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49413.log.gz 2024-09-15T01:12:46.738 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61157.log 2024-09-15T01:12:46.739 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55687.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55687.log.gz 2024-09-15T01:12:46.739 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61383.log 2024-09-15T01:12:46.740 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61157.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61157.log.gz 2024-09-15T01:12:46.740 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.39686.log.gz 2024-09-15T01:12:46.740 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71435.log 2024-09-15T01:12:46.742 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61383.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61383.log.gz 2024-09-15T01:12:46.742 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69130.log 2024-09-15T01:12:46.742 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71435.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71435.log.gz 2024-09-15T01:12:46.750 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45006.log.gz 2024-09-15T01:12:46.751 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62860.log 2024-09-15T01:12:46.752 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69130.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69130.log.gz 2024-09-15T01:12:46.752 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36416.log 2024-09-15T01:12:46.754 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62860.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62860.log.gz 2024-09-15T01:12:46.754 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48517.log 2024-09-15T01:12:46.755 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.36416.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36416.log.gz 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49758.log.gz 2024-09-15T01:12:46.755 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:46.755 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47808.log 2024-09-15T01:12:46.755 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48517.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59066.log 2024-09-15T01:12:46.757 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47808.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48517.log.gz 2024-09-15T01:12:46.757 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63335.log 2024-09-15T01:12:46.757 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59066.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59066.log.gz 2024-09-15T01:12:46.764 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39251.log 2024-09-15T01:12:46.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63335.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63335.log.gz 2024-09-15T01:12:46.773 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66542.log 2024-09-15T01:12:46.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61028.log 2024-09-15T01:12:46.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66542.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66542.log.gz 2024-09-15T01:12:46.774 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54222.log 2024-09-15T01:12:46.783 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61028.log: 83.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39251.log.gz 2024-09-15T01:12:46.790 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61028.log.gz 2024-09-15T01:12:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49816.log 2024-09-15T01:12:46.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54222.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54222.log.gz 2024-09-15T01:12:46.794 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55401.log 2024-09-15T01:12:46.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49816.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49816.log.gz 2024-09-15T01:12:46.795 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49549.log 2024-09-15T01:12:46.805 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42177.log 2024-09-15T01:12:46.813 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49549.log: 87.5% 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55401.log.gz 2024-09-15T01:12:46.813 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49549.log.gz 2024-09-15T01:12:46.814 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65242.log 2024-09-15T01:12:46.814 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42177.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42177.log.gz 2024-09-15T01:12:46.819 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48581.log 2024-09-15T01:12:46.822 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65242.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65242.log.gz 2024-09-15T01:12:46.830 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41138.log 2024-09-15T01:12:46.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48581.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48581.log.gz 2024-09-15T01:12:46.831 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57311.log 2024-09-15T01:12:46.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43834.log 2024-09-15T01:12:46.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57311.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57311.log.gz 2024-09-15T01:12:46.833 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55131.log 2024-09-15T01:12:46.835 INFO:teuthology.orchestra.run.smithi059.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41138.log.gz 2024-09-15T01:12:46.842 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52962.log 2024-09-15T01:12:46.850 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55131.log: /var/log/ceph/cluster2-client.mirror.43834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61404.log 2024-09-15T01:12:46.852 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52962.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55131.log.gz 2024-09-15T01:12:46.853 INFO:teuthology.orchestra.run.smithi059.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41741.log.gz 2024-09-15T01:12:46.853 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43834.log.gz 2024-09-15T01:12:46.854 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52962.log.gz 2024-09-15T01:12:46.854 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52802.log 2024-09-15T01:12:46.863 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61404.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61404.log.gz 2024-09-15T01:12:46.863 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66632.log 2024-09-15T01:12:46.872 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52802.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52802.log.gz 2024-09-15T01:12:46.873 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56451.log 2024-09-15T01:12:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66632.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66632.log.gz 2024-09-15T01:12:46.896 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62942.log 2024-09-15T01:12:46.898 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56451.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56451.log.gz 2024-09-15T01:12:46.901 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65408.log 2024-09-15T01:12:46.903 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62942.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62942.log.gz 2024-09-15T01:12:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72593.log 2024-09-15T01:12:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65408.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65408.log.gz 2024-09-15T01:12:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46937.log 2024-09-15T01:12:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72593.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72593.log.gz 2024-09-15T01:12:46.908 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46937.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46916.log.gz 2024-09-15T01:12:46.909 INFO:teuthology.orchestra.run.smithi059.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46937.log.gz 2024-09-15T01:12:46.909 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53388.log 2024-09-15T01:12:46.911 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53388.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53388.log.gz 2024-09-15T01:12:46.912 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48989.log.gz 2024-09-15T01:12:46.913 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67734.log 2024-09-15T01:12:46.915 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60803.log 2024-09-15T01:12:46.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38723.log 2024-09-15T01:12:46.915 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67734.log.gz 2024-09-15T01:12:46.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40346.log 2024-09-15T01:12:46.915 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60803.log.gz 2024-09-15T01:12:46.915 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44656.log 2024-09-15T01:12:46.919 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.38723.log: /var/log/ceph/cluster2-client.mirror.40346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68848.log 2024-09-15T01:12:46.919 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59090.log 2024-09-15T01:12:46.919 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68848.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44656.log.gz 2024-09-15T01:12:46.919 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68848.log.gz 2024-09-15T01:12:46.919 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56206.log 2024-09-15T01:12:46.920 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38723.log.gz 2024-09-15T01:12:46.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54425.log 2024-09-15T01:12:46.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56206.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59090.log.gz 2024-09-15T01:12:46.920 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38956.log 2024-09-15T01:12:46.924 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54425.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56206.log.gz 2024-09-15T01:12:46.924 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63004.log 2024-09-15T01:12:46.924 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.38956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61608.log 2024-09-15T01:12:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63004.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54425.log.gz 2024-09-15T01:12:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58749.log 2024-09-15T01:12:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61608.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39452.log 2024-09-15T01:12:46.925 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61608.log.gz 2024-09-15T01:12:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58749.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49132.log 2024-09-15T01:12:46.925 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63004.log.gz 2024-09-15T01:12:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58749.log.gz 2024-09-15T01:12:46.925 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.37422.log 2024-09-15T01:12:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49132.log: /var/log/ceph/cluster2-client.mirror.39452.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49132.log.gz 2024-09-15T01:12:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42116.log 2024-09-15T01:12:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60707.log 2024-09-15T01:12:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42116.log: /var/log/ceph/cluster2-client.mirror.37422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55844.log 2024-09-15T01:12:46.930 INFO:teuthology.orchestra.run.smithi059.stderr: 90.0%/var/log/ceph/cluster1-client.mirror.60707.log: -- replaced with /var/log/ceph/cluster2-client.mirror.42116.log.gz 2024-09-15T01:12:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60707.log.gz 2024-09-15T01:12:46.930 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54019.log 2024-09-15T01:12:46.930 INFO:teuthology.orchestra.run.smithi059.stderr: 77.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38956.log.gz 2024-09-15T01:12:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.55844.log: -- /var/log/ceph/cluster2-client.mirror.68622.log 2024-09-15T01:12:46.931 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64412.log 2024-09-15T01:12:46.931 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68622.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55844.log.gz 2024-09-15T01:12:46.931 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65594.log 2024-09-15T01:12:46.931 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64412.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37422.log.gz 2024-09-15T01:12:46.931 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68622.log.gz 2024-09-15T01:12:46.936 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54019.log.gz 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62212.log 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64412.log.gz 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose/var/log/ceph/cluster2-client.mirror.65594.log: -- /var/log/ceph/cluster1-client.mirror.57376.log 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60414.log 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65594.log.gz 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57376.log: /var/log/ceph/cluster1-client.mirror.62212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47670.log 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60414.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62212.log.gz 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60414.log.gz 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57376.log.gz 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64681.log 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44241.log 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47670.log.gz 2024-09-15T01:12:46.937 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62413.log 2024-09-15T01:12:46.938 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62798.log 2024-09-15T01:12:46.938 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64681.log.gz 2024-09-15T01:12:46.940 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.62413.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62413.log.gz 2024-09-15T01:12:46.940 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42642.log 2024-09-15T01:12:46.940 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62798.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62798.log.gz 2024-09-15T01:12:46.940 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44496.log 2024-09-15T01:12:46.940 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49174.log 2024-09-15T01:12:46.941 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44496.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44496.log.gz 2024-09-15T01:12:46.941 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51728.log 2024-09-15T01:12:46.942 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49174.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44241.log.gz 2024-09-15T01:12:46.942 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62569.log 2024-09-15T01:12:46.944 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51728.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51728.log.gz 2024-09-15T01:12:46.945 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50687.log 2024-09-15T01:12:46.951 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62569.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42642.log.gz 2024-09-15T01:12:46.956 INFO:teuthology.orchestra.run.smithi059.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62569.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55798.log 2024-09-15T01:12:46.957 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50687.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50687.log.gz 2024-09-15T01:12:46.957 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60040.log 2024-09-15T01:12:46.958 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55798.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55798.log.gz 2024-09-15T01:12:46.958 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:46.958 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66092.log 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60040.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68585.log 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66092.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60040.log.gz 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66092.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40161.log 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68027.log 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68585.log.gz 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52085.log 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68027.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68027.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47966.log 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40161.log.gz 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52085.log: 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58458.log 2024-09-15T01:12:46.963 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52085.log.gz 2024-09-15T01:12:46.964 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48841.log 2024-09-15T01:12:46.965 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58458.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58458.log.gz 2024-09-15T01:12:46.973 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45932.log 2024-09-15T01:12:46.974 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48841.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70116.log.gz 2024-09-15T01:12:46.975 INFO:teuthology.orchestra.run.smithi059.stderr: 85.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48841.log.gz 2024-09-15T01:12:46.975 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61137.log 2024-09-15T01:12:46.976 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45932.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45932.log.gz 2024-09-15T01:12:46.976 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71929.log 2024-09-15T01:12:46.979 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61137.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63294.log 2024-09-15T01:12:46.980 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71929.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61137.log.gz 2024-09-15T01:12:46.980 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71929.log.gz 2024-09-15T01:12:46.980 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69324.log 2024-09-15T01:12:46.980 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.69533.log 2024-09-15T01:12:46.980 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63294.log.gz 2024-09-15T01:12:46.980 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69324.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69324.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43001.log 2024-09-15T01:12:46.980 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:46.980 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.1.69533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46571.log 2024-09-15T01:12:46.981 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.43001.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.43001.log.gz 2024-09-15T01:12:46.981 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41483.log 2024-09-15T01:12:46.985 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46571.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.69533.log.gz 2024-09-15T01:12:46.985 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41319.log 2024-09-15T01:12:46.985 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.41483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65138.log 2024-09-15T01:12:46.986 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52536.log 2024-09-15T01:12:46.987 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65138.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.41483.log.gz 2024-09-15T01:12:46.990 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65138.log.gz 2024-09-15T01:12:46.990 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67869.log 2024-09-15T01:12:46.991 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52536.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52536.log.gz 2024-09-15T01:12:46.991 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61963.log 2024-09-15T01:12:46.991 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67869.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67869.log.gz 2024-09-15T01:12:46.993 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51587.log 2024-09-15T01:12:46.993 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61963.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61963.log.gz 2024-09-15T01:12:46.993 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40346.log.gz 2024-09-15T01:12:46.993 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61560.log 2024-09-15T01:12:46.994 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51587.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51587.log.gz 2024-09-15T01:12:46.994 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70951.log 2024-09-15T01:12:46.995 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61560.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61560.log.gz 2024-09-15T01:12:46.997 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48278.log 2024-09-15T01:12:46.998 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70951.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70951.log.gz 2024-09-15T01:12:46.998 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66339.log 2024-09-15T01:12:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48278.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48278.log.gz 2024-09-15T01:12:47.001 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64020.log 2024-09-15T01:12:47.001 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66339.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66339.log.gz 2024-09-15T01:12:47.001 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39100.log 2024-09-15T01:12:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64020.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64020.log.gz 2024-09-15T01:12:47.003 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56338.log 2024-09-15T01:12:47.005 INFO:teuthology.orchestra.run.smithi059.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39452.log.gz 2024-09-15T01:12:47.006 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57809.log 2024-09-15T01:12:47.007 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56338.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56338.log.gz 2024-09-15T01:12:47.007 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53886.log 2024-09-15T01:12:47.008 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57809.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57809.log.gz 2024-09-15T01:12:47.012 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39823.log 2024-09-15T01:12:47.016 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53886.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53886.log.gz 2024-09-15T01:12:47.017 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64722.log 2024-09-15T01:12:47.020 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50503.log 2024-09-15T01:12:47.027 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64722.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64722.log.gz 2024-09-15T01:12:47.030 INFO:teuthology.orchestra.run.smithi059.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39100.log.gz 2024-09-15T01:12:47.030 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67194.log 2024-09-15T01:12:47.033 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50503.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46412.log.gz 2024-09-15T01:12:47.034 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48782.log 2024-09-15T01:12:47.034 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67194.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67194.log.gz 2024-09-15T01:12:47.035 INFO:teuthology.orchestra.run.smithi059.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50503.log.gz 2024-09-15T01:12:47.036 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39734.log 2024-09-15T01:12:47.037 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59613.log 2024-09-15T01:12:47.038 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.39734.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48782.log.gz 2024-09-15T01:12:47.038 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63438.log 2024-09-15T01:12:47.039 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54633.log 2024-09-15T01:12:47.039 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63438.log: /var/log/ceph/cluster1-client.mirror.59613.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59613.log.gz 2024-09-15T01:12:47.040 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72049.log 2024-09-15T01:12:47.041 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54633.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39734.log.gz 2024-09-15T01:12:47.042 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54633.log.gz 2024-09-15T01:12:47.042 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60932.log 2024-09-15T01:12:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72049.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72049.log.gz 2024-09-15T01:12:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56906.log 2024-09-15T01:12:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60932.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60932.log.gz 2024-09-15T01:12:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37439.log 2024-09-15T01:12:47.047 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56906.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56906.log.gz 2024-09-15T01:12:47.047 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70604.log 2024-09-15T01:12:47.047 INFO:teuthology.orchestra.run.smithi059.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63438.log.gz 2024-09-15T01:12:47.048 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52332.log 2024-09-15T01:12:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70604.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70604.log.gz 2024-09-15T01:12:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63916.log 2024-09-15T01:12:47.049 INFO:teuthology.orchestra.run.smithi059.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37439.log.gz 2024-09-15T01:12:47.052 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52332.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52332.log.gz 2024-09-15T01:12:47.052 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48724.log 2024-09-15T01:12:47.053 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63916.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39823.log.gz 2024-09-15T01:12:47.054 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63916.log.gz 2024-09-15T01:12:47.055 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36290.log 2024-09-15T01:12:47.055 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72025.log 2024-09-15T01:12:47.056 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48724.log: /var/log/ceph/cluster1-client.admin.36290.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36290.log.gz 2024-09-15T01:12:47.065 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67802.log 2024-09-15T01:12:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.72025.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72025.log.gz 2024-09-15T01:12:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-15T01:12:47.075 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67802.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67802.log.gz 2024-09-15T01:12:47.075 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48420.log 2024-09-15T01:12:47.078 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47808.log.gz 2024-09-15T01:12:47.080 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54991.log 2024-09-15T01:12:47.080 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48420.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48420.log.gz 2024-09-15T01:12:47.080 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64288.log 2024-09-15T01:12:47.081 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54991.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54991.log.gz 2024-09-15T01:12:47.082 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49231.log 2024-09-15T01:12:47.083 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64288.log: /var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58365.log 2024-09-15T01:12:47.084 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49231.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49231.log.gz 2024-09-15T01:12:47.084 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62482.log 2024-09-15T01:12:47.085 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46529.log 2024-09-15T01:12:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63149.log 2024-09-15T01:12:47.087 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64288.log.gz 2024-09-15T01:12:47.087 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62482.log: /var/log/ceph/cluster1-client.mirror.58365.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58365.log.gz 2024-09-15T01:12:47.089 INFO:teuthology.orchestra.run.smithi059.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62482.log.gz 2024-09-15T01:12:47.090 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46529.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46529.log.gz 2024-09-15T01:12:47.090 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.37626.log 2024-09-15T01:12:47.093 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36862.log 2024-09-15T01:12:47.093 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.1.37626.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63149.log.gz 2024-09-15T01:12:47.093 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43112.log 2024-09-15T01:12:47.093 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.36862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42957.log 2024-09-15T01:12:47.094 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43112.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43112.log.gz 2024-09-15T01:12:47.094 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63604.log 2024-09-15T01:12:47.097 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42957.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42957.log.gz 2024-09-15T01:12:47.097 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64821.log 2024-09-15T01:12:47.097 INFO:teuthology.orchestra.run.smithi059.stderr: 73.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36862.log.gz 2024-09-15T01:12:47.097 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58887.log 2024-09-15T01:12:47.098 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58500.log 2024-09-15T01:12:47.098 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63604.log.gz 2024-09-15T01:12:47.099 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58887.log: /var/log/ceph/cluster2-client.mirror.64821.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58887.log.gz 2024-09-15T01:12:47.100 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43420.log 2024-09-15T01:12:47.101 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58500.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58500.log.gz 2024-09-15T01:12:47.103 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41319.log.gz 2024-09-15T01:12:47.103 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48745.log 2024-09-15T01:12:47.103 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43420.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43420.log.gz 2024-09-15T01:12:47.105 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59676.log 2024-09-15T01:12:47.105 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48745.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48745.log.gz 2024-09-15T01:12:47.107 INFO:teuthology.orchestra.run.smithi059.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64821.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.36782.log 2024-09-15T01:12:47.107 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:47.107 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63108.log 2024-09-15T01:12:47.107 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59676.log: /var/log/ceph/cluster2-client.mirror.1.36782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56789.log 2024-09-15T01:12:47.109 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63108.log: 89.7% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63108.log.gz 2024-09-15T01:12:47.109 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.59676.log.gz 2024-09-15T01:12:47.109 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51707.log 2024-09-15T01:12:47.110 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46491.log.gz/var/log/ceph/cluster1-client.mirror.56789.log: 2024-09-15T01:12:47.110 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56789.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48500.log 2024-09-15T01:12:47.110 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:47.110 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68094.log 2024-09-15T01:12:47.110 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48500.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51707.log.gz 2024-09-15T01:12:47.110 INFO:teuthology.orchestra.run.smithi059.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48500.log.gz 2024-09-15T01:12:47.111 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59203.log 2024-09-15T01:12:47.111 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/cluster1-client.mirror.68094.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71479.log 2024-09-15T01:12:47.111 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68094.log.gz 2024-09-15T01:12:47.113 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59203.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59203.log.gz 2024-09-15T01:12:47.113 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64536.log 2024-09-15T01:12:47.114 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40657.log 2024-09-15T01:12:47.115 INFO:teuthology.orchestra.run.smithi059.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71479.log.gz 2024-09-15T01:12:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40657.log: gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.64536.log: /var/log/ceph/cluster1-client.mirror.55308.log 2024-09-15T01:12:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54609.log 2024-09-15T01:12:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55308.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55308.log.gz 2024-09-15T01:12:47.119 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40657.log.gz 2024-09-15T01:12:47.119 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.57038.log 2024-09-15T01:12:47.119 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63999.log 2024-09-15T01:12:47.119 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64536.log.gz 2024-09-15T01:12:47.119 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43490.log 2024-09-15T01:12:47.119 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57038.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54609.log.gz 2024-09-15T01:12:47.120 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57086.log 2024-09-15T01:12:47.120 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57038.log.gz 2024-09-15T01:12:47.120 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52986.log 2024-09-15T01:12:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57086.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57086.log.gz 2024-09-15T01:12:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70724.log 2024-09-15T01:12:47.122 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63999.log.gz 2024-09-15T01:12:47.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46390.log 2024-09-15T01:12:47.123 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52986.log.gz 2024-09-15T01:12:47.124 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46469.log 2024-09-15T01:12:47.125 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46390.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46390.log.gz 2024-09-15T01:12:47.125 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61446.log 2024-09-15T01:12:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46469.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43490.log.gz 2024-09-15T01:12:47.128 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46469.log.gz 2024-09-15T01:12:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45651.log 2024-09-15T01:12:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61446.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45793.log 2024-09-15T01:12:47.128 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70724.log.gz 2024-09-15T01:12:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45651.log: gzip 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45651.log.gz 2024-09-15T01:12:47.129 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68564.log 2024-09-15T01:12:47.129 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61446.log.gz 2024-09-15T01:12:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53838.log 2024-09-15T01:12:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45793.log: /var/log/ceph/cluster2-client.mirror.68564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68193.log 2024-09-15T01:12:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53838.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53838.log.gz 2024-09-15T01:12:47.131 INFO:teuthology.orchestra.run.smithi059.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68564.log.gz 2024-09-15T01:12:47.131 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47339.log 2024-09-15T01:12:47.133 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68193.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68193.log.gz 2024-09-15T01:12:47.133 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71879.log/var/log/ceph/cluster2-client.mirror.47339.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47339.log.gz 2024-09-15T01:12:47.133 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:47.134 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43705.log 2024-09-15T01:12:47.135 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71879.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71879.log.gz 2024-09-15T01:12:47.135 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66249.log 2024-09-15T01:12:47.141 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43705.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43705.log.gz 2024-09-15T01:12:47.148 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72717.log 2024-09-15T01:12:47.148 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66249.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66249.log.gz 2024-09-15T01:12:47.148 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42851.log 2024-09-15T01:12:47.148 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72717.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72717.log.gz 2024-09-15T01:12:47.148 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60506.log 2024-09-15T01:12:47.148 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42851.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42851.log.gz 2024-09-15T01:12:47.148 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54288.log 2024-09-15T01:12:47.148 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37456.log 2024-09-15T01:12:47.149 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54288.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54288.log.gz 2024-09-15T01:12:47.149 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37217.log 2024-09-15T01:12:47.149 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.37456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70687.log 2024-09-15T01:12:47.149 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.37217.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60506.log.gz 2024-09-15T01:12:47.157 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41182.log 2024-09-15T01:12:47.157 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70687.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70687.log.gz 2024-09-15T01:12:47.157 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51209.log 2024-09-15T01:12:47.157 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67577.log 2024-09-15T01:12:47.157 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51816.log 2024-09-15T01:12:47.157 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51209.log.gz 2024-09-15T01:12:47.157 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67577.log: 56.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37456.log.gz 2024-09-15T01:12:47.157 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67577.log.gz 2024-09-15T01:12:47.157 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44836.log 2024-09-15T01:12:47.157 INFO:teuthology.orchestra.run.smithi059.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37217.log.gz 2024-09-15T01:12:47.158 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42338.log 2024-09-15T01:12:47.158 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51816.log.gz 2024-09-15T01:12:47.158 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37183.log 2024-09-15T01:12:47.158 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42338.log: 89.4% 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42338.log.gz 2024-09-15T01:12:47.158 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44836.log.gz 2024-09-15T01:12:47.158 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65957.log 2024-09-15T01:12:47.158 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62756.log 2024-09-15T01:12:47.158 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.65957.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65957.log.gz 2024-09-15T01:12:47.158 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41182.log.gz 2024-09-15T01:12:47.159 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.37183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-15T01:12:47.160 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62756.log: gzip -5 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37183.log.gz 2024-09-15T01:12:47.161 INFO:teuthology.orchestra.run.smithi059.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62756.log.gz 2024-09-15T01:12:47.161 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-osd.1.log: --verbose -- /var/log/ceph/cluster1-client.mirror.56636.log 2024-09-15T01:12:47.164 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56636.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56636.log.gz 2024-09-15T01:12:47.164 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53320.log 2024-09-15T01:12:47.164 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53589.log 2024-09-15T01:12:47.164 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.53320.log: -- /var/log/ceph/cluster1-client.mirror.51908.log 2024-09-15T01:12:47.165 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53320.log.gz 2024-09-15T01:12:47.165 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53589.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53589.log.gz 2024-09-15T01:12:47.168 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51908.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51908.log.gz 2024-09-15T01:12:47.168 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43092.log 2024-09-15T01:12:47.168 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68927.log 2024-09-15T01:12:47.168 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40323.log 2024-09-15T01:12:47.169 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68927.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68927.log.gz 2024-09-15T01:12:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43092.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43092.log.gz 2024-09-15T01:12:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53457.log 2024-09-15T01:12:47.173 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.40323.log: /var/log/ceph/cluster1-client.mirror.59540.log 2024-09-15T01:12:47.173 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41254.log 2024-09-15T01:12:47.173 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59540.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59540.log.gz 2024-09-15T01:12:47.173 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67644.log 2024-09-15T01:12:47.174 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63772.log 2024-09-15T01:12:47.174 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67644.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67644.log.gz 2024-09-15T01:12:47.176 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51977.log 2024-09-15T01:12:47.176 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63772.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53457.log.gz 2024-09-15T01:12:47.178 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41254.log.gz 2024-09-15T01:12:47.179 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63772.log.gz 2024-09-15T01:12:47.179 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66384.log 2024-09-15T01:12:47.182 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51977.log: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68544.log 2024-09-15T01:12:47.182 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.51977.log.gz 2024-09-15T01:12:47.182 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62963.log 2024-09-15T01:12:47.182 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68544.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68544.log.gz 2024-09-15T01:12:47.182 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66384.log.gz 2024-09-15T01:12:47.183 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57131.log 2024-09-15T01:12:47.183 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62963.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62963.log.gz 2024-09-15T01:12:47.183 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45912.log 2024-09-15T01:12:47.184 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35700.log 2024-09-15T01:12:47.185 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45912.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45912.log.gz 2024-09-15T01:12:47.187 INFO:teuthology.orchestra.run.smithi059.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57131.log.gz 2024-09-15T01:12:47.187 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45589.log 2024-09-15T01:12:47.187 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35700.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35700.log.gz 2024-09-15T01:12:47.189 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47036.log 2024-09-15T01:12:47.189 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44984.log 2024-09-15T01:12:47.189 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47036.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45589.log.gz 2024-09-15T01:12:47.190 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49026.log 2024-09-15T01:12:47.190 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44984.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44984.log.gz 2024-09-15T01:12:47.190 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70277.log 2024-09-15T01:12:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.49026.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49026.log.gz 2024-09-15T01:12:47.193 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47036.log.gz 2024-09-15T01:12:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37302.log 2024-09-15T01:12:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49290.log 2024-09-15T01:12:47.195 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.37302.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70277.log.gz 2024-09-15T01:12:47.195 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37302.log.gz 2024-09-15T01:12:47.195 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52352.log 2024-09-15T01:12:47.195 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50346.log 2024-09-15T01:12:47.195 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49290.log.gz 2024-09-15T01:12:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50346.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50346.log.gz 2024-09-15T01:12:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63562.log 2024-09-15T01:12:47.199 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52352.log.gz 2024-09-15T01:12:47.202 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63562.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63562.log.gz 2024-09-15T01:12:47.202 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/cluster2-client.mirror.62922.log 2024-09-15T01:12:47.202 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56185.log 2024-09-15T01:12:47.203 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62922.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62922.log.gz 2024-09-15T01:12:47.204 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49647.log 2024-09-15T01:12:47.205 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56185.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56185.log.gz 2024-09-15T01:12:47.206 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40585.log 2024-09-15T01:12:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43682.log 2024-09-15T01:12:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49647.log: 81.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49647.log.gz 2024-09-15T01:12:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47238.log 2024-09-15T01:12:47.211 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43682.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40585.log.gz 2024-09-15T01:12:47.211 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65097.log 2024-09-15T01:12:47.212 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43682.log.gz 2024-09-15T01:12:47.214 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65097.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65097.log.gz 2024-09-15T01:12:47.215 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63376.log 2024-09-15T01:12:47.219 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60683.log 2024-09-15T01:12:47.222 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63376.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63376.log.gz 2024-09-15T01:12:47.224 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52689.log 2024-09-15T01:12:47.227 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60683.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40323.log.gz 2024-09-15T01:12:47.227 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60683.log.gz 2024-09-15T01:12:47.229 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70704.log 2024-09-15T01:12:47.233 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52689.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52689.log.gz 2024-09-15T01:12:47.236 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57737.log 2024-09-15T01:12:47.241 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70704.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70704.log.gz 2024-09-15T01:12:47.241 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49593.log 2024-09-15T01:12:47.254 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57737.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57737.log.gz 2024-09-15T01:12:47.254 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61091.log 2024-09-15T01:12:47.254 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49593.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49593.log.gz 2024-09-15T01:12:47.256 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61091.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61091.log.gz 2024-09-15T01:12:47.256 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50888.log 2024-09-15T01:12:47.258 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69951.log 2024-09-15T01:12:47.258 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50888.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50888.log.gz 2024-09-15T01:12:47.258 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39600.log 2024-09-15T01:12:47.258 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69951.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69951.log.gz 2024-09-15T01:12:47.263 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47626.log 2024-09-15T01:12:47.263 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48130.log 2024-09-15T01:12:47.264 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47626.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47626.log.gz 2024-09-15T01:12:47.267 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67599.log 2024-09-15T01:12:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48130.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39600.log.gz 2024-09-15T01:12:47.285 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46172.log 2024-09-15T01:12:47.286 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67599.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67599.log.gz 2024-09-15T01:12:47.300 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47099.log 2024-09-15T01:12:47.300 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46172.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46172.log.gz 2024-09-15T01:12:47.300 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49450.log 2024-09-15T01:12:47.306 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47099.log: 80.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43468.log 2024-09-15T01:12:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47099.log.gz 2024-09-15T01:12:47.312 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49450.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49450.log.gz 2024-09-15T01:12:47.326 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56314.log 2024-09-15T01:12:47.326 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43468.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43468.log.gz 2024-09-15T01:12:47.340 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69301.log 2024-09-15T01:12:47.340 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56314.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56314.log.gz 2024-09-15T01:12:47.349 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62502.log 2024-09-15T01:12:47.349 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69301.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69301.log.gz 2024-09-15T01:12:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48377.log 2024-09-15T01:12:47.361 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.62502.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62502.log.gz 2024-09-15T01:12:47.369 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54858.log 2024-09-15T01:12:47.369 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48377.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48377.log.gz 2024-09-15T01:12:47.379 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49838.log.gz 2024-09-15T01:12:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45527.log 2024-09-15T01:12:47.381 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54858.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54858.log.gz 2024-09-15T01:12:47.381 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47296.log 2024-09-15T01:12:47.383 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45527.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45527.log.gz 2024-09-15T01:12:47.383 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60016.log 2024-09-15T01:12:47.384 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47296.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47296.log.gz 2024-09-15T01:12:47.384 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49779.log 2024-09-15T01:12:47.385 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60016.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60016.log.gz 2024-09-15T01:12:47.386 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65470.log 2024-09-15T01:12:47.386 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49779.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49779.log.gz 2024-09-15T01:12:47.386 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41672.log 2024-09-15T01:12:47.387 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65470.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65470.log.gz 2024-09-15T01:12:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42383.log 2024-09-15T01:12:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57962.log 2024-09-15T01:12:47.389 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42383.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41672.log.gz 2024-09-15T01:12:47.395 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42383.log.gz 2024-09-15T01:12:47.395 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40633.log 2024-09-15T01:12:47.396 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57962.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57962.log.gz 2024-09-15T01:12:47.396 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44155.log 2024-09-15T01:12:47.397 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72341.log 2024-09-15T01:12:47.402 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44155.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40633.log.gz 2024-09-15T01:12:47.403 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44155.log.gz 2024-09-15T01:12:47.404 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39364.log 2024-09-15T01:12:47.404 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72341.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72341.log.gz 2024-09-15T01:12:47.406 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65889.log 2024-09-15T01:12:47.406 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49310.log 2024-09-15T01:12:47.407 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.65889.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65889.log.gz 2024-09-15T01:12:47.407 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63661.log 2024-09-15T01:12:47.407 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49310.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49310.log.gz 2024-09-15T01:12:47.408 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39364.log.gz 2024-09-15T01:12:47.408 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50366.log 2024-09-15T01:12:47.409 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.63661.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63661.log.gz 2024-09-15T01:12:47.409 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37354.log 2024-09-15T01:12:47.410 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50366.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50366.log.gz 2024-09-15T01:12:47.410 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48762.log 2024-09-15T01:12:47.411 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.37354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65821.log 2024-09-15T01:12:47.412 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48762.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48762.log.gz 2024-09-15T01:12:47.413 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53007.log 2024-09-15T01:12:47.415 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65821.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.37354.log.gz 2024-09-15T01:12:47.415 INFO:teuthology.orchestra.run.smithi059.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65821.log.gz 2024-09-15T01:12:47.416 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69753.log 2024-09-15T01:12:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53007.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53007.log.gz 2024-09-15T01:12:47.419 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57488.log 2024-09-15T01:12:47.419 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69753.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69753.log.gz 2024-09-15T01:12:47.420 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71979.log 2024-09-15T01:12:47.420 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57488.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57488.log.gz 2024-09-15T01:12:47.421 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59315.log 2024-09-15T01:12:47.422 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71979.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71979.log.gz 2024-09-15T01:12:47.422 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48724.log.gz 2024-09-15T01:12:47.422 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48088.log 2024-09-15T01:12:47.423 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.59315.log: -- /var/log/ceph/cluster1-client.mirror.53139.log 2024-09-15T01:12:47.424 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59315.log.gz 2024-09-15T01:12:47.424 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48088.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48088.log.gz 2024-09-15T01:12:47.424 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68762.log 2024-09-15T01:12:47.425 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53139.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53139.log.gz 2024-09-15T01:12:47.425 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52065.log 2024-09-15T01:12:47.426 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69641.log 2024-09-15T01:12:47.427 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68762.log: /var/log/ceph/cluster2-client.mirror.52065.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68762.log.gz 2024-09-15T01:12:47.427 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56362.log 2024-09-15T01:12:47.427 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52065.log.gz 2024-09-15T01:12:47.429 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69641.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69641.log.gz 2024-09-15T01:12:47.429 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60345.log 2024-09-15T01:12:47.430 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56362.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56362.log.gz 2024-09-15T01:12:47.431 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64040.log 2024-09-15T01:12:47.431 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51860.log 2024-09-15T01:12:47.433 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64040.log: /var/log/ceph/cluster1-client.mirror.60345.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60345.log.gz 2024-09-15T01:12:47.433 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66024.log 2024-09-15T01:12:47.434 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51860.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51860.log.gz 2024-09-15T01:12:47.440 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68004.log 2024-09-15T01:12:47.440 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49174.log.gz 2024-09-15T01:12:47.440 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67689.log 2024-09-15T01:12:47.440 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68004.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66024.log.gz 2024-09-15T01:12:47.440 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64495.log 2024-09-15T01:12:47.440 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.68004.log.gz 2024-09-15T01:12:47.440 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67689.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67689.log.gz 2024-09-15T01:12:47.441 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64040.log.gz 2024-09-15T01:12:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67847.log 2024-09-15T01:12:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47562.log 2024-09-15T01:12:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64495.log: /var/log/ceph/cluster1-client.mirror.67847.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67847.log.gz 2024-09-15T01:12:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66047.log 2024-09-15T01:12:47.441 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3%/var/log/ceph/cluster2-client.mirror.47562.log: -- replaced with /var/log/ceph/cluster2-client.mirror.64495.log.gz 2024-09-15T01:12:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39547.log 2024-09-15T01:12:47.443 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66047.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66047.log.gz 2024-09-15T01:12:47.443 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54696.log 2024-09-15T01:12:47.443 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.39547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64764.log 2024-09-15T01:12:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40282.log 2024-09-15T01:12:47.444 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54696.log.gz 2024-09-15T01:12:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64764.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64764.log.gz 2024-09-15T01:12:47.445 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49332.log 2024-09-15T01:12:47.447 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40282.log: gzip/var/log/ceph/cluster2-client.mirror.49332.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.39547.log.gz 2024-09-15T01:12:47.448 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52870.log 2024-09-15T01:12:47.448 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40282.log.gz 2024-09-15T01:12:47.452 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50414.log 2024-09-15T01:12:47.457 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52870.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52870.log.gz 2024-09-15T01:12:47.459 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62839.log 2024-09-15T01:12:47.460 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50414.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50414.log.gz 2024-09-15T01:12:47.461 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42803.log 2024-09-15T01:12:47.462 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62839.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62839.log.gz 2024-09-15T01:12:47.465 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43296.log 2024-09-15T01:12:47.466 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42803.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42803.log.gz 2024-09-15T01:12:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44413.log 2024-09-15T01:12:47.476 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43296.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43296.log.gz 2024-09-15T01:12:47.478 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52914.log 2024-09-15T01:12:47.487 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54810.log 2024-09-15T01:12:47.497 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52914.log: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54923.log 2024-09-15T01:12:47.501 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54810.log: -- replaced with /var/log/ceph/cluster1-client.mirror.52914.log.gz 2024-09-15T01:12:47.506 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57398.log 2024-09-15T01:12:47.510 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54923.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54810.log.gz 2024-09-15T01:12:47.520 INFO:teuthology.orchestra.run.smithi059.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54923.log.gz 2024-09-15T01:12:47.524 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56001.log 2024-09-15T01:12:47.525 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57398.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44413.log.gz 2024-09-15T01:12:47.526 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57398.log.gz 2024-09-15T01:12:47.531 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46092.log 2024-09-15T01:12:47.536 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56001.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45793.log.gz 2024-09-15T01:12:47.542 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57713.log 2024-09-15T01:12:47.544 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46092.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46092.log.gz 2024-09-15T01:12:47.544 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42514.log 2024-09-15T01:12:47.544 INFO:teuthology.orchestra.run.smithi059.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56001.log.gz 2024-09-15T01:12:47.544 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43054.log 2024-09-15T01:12:47.544 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.42514.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57713.log.gz 2024-09-15T01:12:47.545 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58437.log 2024-09-15T01:12:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.43054.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43054.log.gz 2024-09-15T01:12:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48340.log 2024-09-15T01:12:47.550 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.42514.log.gz 2024-09-15T01:12:47.550 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39710.log 2024-09-15T01:12:47.550 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48340.log: 87.7% 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48340.log.gz 2024-09-15T01:12:47.550 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58437.log.gz 2024-09-15T01:12:47.550 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63583.log 2024-09-15T01:12:47.550 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.38956.log 2024-09-15T01:12:47.551 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53477.log 2024-09-15T01:12:47.552 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63583.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63583.log.gz 2024-09-15T01:12:47.556 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.2.38956.log: /var/log/ceph/cluster1-client.mirror.39710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67262.log 2024-09-15T01:12:47.556 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44576.log 2024-09-15T01:12:47.560 INFO:teuthology.orchestra.run.smithi059.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53477.log.gz 2024-09-15T01:12:47.560 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39710.log.gz 2024-09-15T01:12:47.560 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44964.log 2024-09-15T01:12:47.560 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44576.log.gz 2024-09-15T01:12:47.560 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46189.log 2024-09-15T01:12:47.561 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44964.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67262.log.gz 2024-09-15T01:12:47.561 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44964.log.gz 2024-09-15T01:12:47.561 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51521.log 2024-09-15T01:12:47.561 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46189.log.gz 2024-09-15T01:12:47.561 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59428.log 2024-09-15T01:12:47.567 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.51521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55581.log 2024-09-15T01:12:47.567 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51521.log.gz 2024-09-15T01:12:47.567 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71257.log 2024-09-15T01:12:47.567 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55581.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55581.log.gz 2024-09-15T01:12:47.567 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35951.log 2024-09-15T01:12:47.567 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71257.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59428.log.gz 2024-09-15T01:12:47.567 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71257.log.gz 2024-09-15T01:12:47.567 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52174.log 2024-09-15T01:12:47.567 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35951.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.35951.log.gz 2024-09-15T01:12:47.567 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61850.log 2024-09-15T01:12:47.568 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70074.log 2024-09-15T01:12:47.568 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.61850.log: /var/log/ceph/cluster2-client.mirror.52174.log: 81.6% 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61850.log.gz 2024-09-15T01:12:47.568 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52174.log.gz 2024-09-15T01:12:47.568 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65035.log 2024-09-15T01:12:47.568 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70074.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70074.log.gz 2024-09-15T01:12:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53436.log 2024-09-15T01:12:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65035.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65035.log.gz 2024-09-15T01:12:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42892.log 2024-09-15T01:12:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53436.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46231.log 2024-09-15T01:12:47.574 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42892.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53436.log.gz 2024-09-15T01:12:47.575 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42892.log.gz 2024-09-15T01:12:47.575 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.64821.log 2024-09-15T01:12:47.582 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62365.log 2024-09-15T01:12:47.596 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.0.64821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56113.log 2024-09-15T01:12:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.62365.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62365.log.gz 2024-09-15T01:12:47.608 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68602.log 2024-09-15T01:12:47.608 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56113.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56113.log.gz 2024-09-15T01:12:47.615 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47966.log.gz 2024-09-15T01:12:47.615 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56226.log 2024-09-15T01:12:47.617 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68602.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68602.log.gz 2024-09-15T01:12:47.621 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44390.log 2024-09-15T01:12:47.634 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56226.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56226.log.gz 2024-09-15T01:12:47.635 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44261.log 2024-09-15T01:12:47.645 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44476.log 2024-09-15T01:12:47.654 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44261.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51884.log 2024-09-15T01:12:47.654 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44476.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46571.log.gz 2024-09-15T01:12:47.655 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44476.log.gz 2024-09-15T01:12:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68804.log 2024-09-15T01:12:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51884.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51884.log.gz 2024-09-15T01:12:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72677.log 2024-09-15T01:12:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68804.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68804.log.gz 2024-09-15T01:12:47.660 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40884.log 2024-09-15T01:12:47.660 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72677.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72677.log.gz 2024-09-15T01:12:47.660 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44390.log.gz 2024-09-15T01:12:47.660 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68385.log 2024-09-15T01:12:47.660 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44261.log.gz 2024-09-15T01:12:47.661 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40884.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36582.log 2024-09-15T01:12:47.662 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45734.log 2024-09-15T01:12:47.662 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.36582.log: /var/log/ceph/cluster2-client.mirror.68385.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68385.log.gz 2024-09-15T01:12:47.664 INFO:teuthology.orchestra.run.smithi059.stderr: 74.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36582.log.gz 2024-09-15T01:12:47.664 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38562.log 2024-09-15T01:12:47.664 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45734.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45734.log.gz 2024-09-15T01:12:47.664 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47460.log 2024-09-15T01:12:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.38562.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41848.log 2024-09-15T01:12:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47460.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47460.log.gz 2024-09-15T01:12:47.666 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42033.log 2024-09-15T01:12:47.667 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54834.log 2024-09-15T01:12:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42033.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38562.log.gz 2024-09-15T01:12:47.671 INFO:teuthology.orchestra.run.smithi059.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.64821.log.gz 2024-09-15T01:12:47.671 INFO:teuthology.orchestra.run.smithi059.stderr: 91.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67059.log 2024-09-15T01:12:47.671 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42033.log.gz/var/log/ceph/cluster1-client.mirror.54834.log: 2024-09-15T01:12:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55864.log 2024-09-15T01:12:47.671 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54834.log.gz 2024-09-15T01:12:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67059.log: gzip -5 --verbose 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67059.log.gz 2024-09-15T01:12:47.671 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/cluster2-client.mirror.44556.log 2024-09-15T01:12:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55864.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55864.log.gz 2024-09-15T01:12:47.674 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45065.log 2024-09-15T01:12:47.674 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53095.log 2024-09-15T01:12:47.674 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45065.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44556.log.gz 2024-09-15T01:12:47.675 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45065.log.gz 2024-09-15T01:12:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61311.log 2024-09-15T01:12:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51499.log 2024-09-15T01:12:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61311.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53095.log.gz 87.6% 2024-09-15T01:12:47.675 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.61311.log.gz 2024-09-15T01:12:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69466.log 2024-09-15T01:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.51499.log: -- /var/log/ceph/cluster1-client.mirror.42362.log 2024-09-15T01:12:47.677 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69466.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51499.log.gz 2024-09-15T01:12:47.677 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59339.log 2024-09-15T01:12:47.679 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42362.log: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63521.log 2024-09-15T01:12:47.679 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42362.log.gz 2024-09-15T01:12:47.679 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56137.log 2024-09-15T01:12:47.679 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63521.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59339.log.gz 2024-09-15T01:12:47.680 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63521.log.gz 2024-09-15T01:12:47.681 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52782.log 2024-09-15T01:12:47.681 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56137.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56137.log.gz 2024-09-15T01:12:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46608.log 2024-09-15T01:12:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52782.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52782.log.gz 2024-09-15T01:12:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47423.log 2024-09-15T01:12:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.46608.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46608.log.gz 2024-09-15T01:12:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59162.log 2024-09-15T01:12:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47423.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47423.log.gz 2024-09-15T01:12:47.687 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62736.log 2024-09-15T01:12:47.687 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.59162.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59162.log.gz 2024-09-15T01:12:47.687 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68502.log 2024-09-15T01:12:47.689 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62736.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62736.log.gz 2024-09-15T01:12:47.689 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52109.log 2024-09-15T01:12:47.689 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68502.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68502.log.gz 2024-09-15T01:12:47.689 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46795.log 2024-09-15T01:12:47.689 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52109.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52109.log.gz 2024-09-15T01:12:47.690 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42314.log 2024-09-15T01:12:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46795.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46795.log.gz 2024-09-15T01:12:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48967.log 2024-09-15T01:12:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.42314.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42314.log.gz 2024-09-15T01:12:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38882.log 2024-09-15T01:12:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48967.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48967.log.gz 2024-09-15T01:12:47.695 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54786.log 2024-09-15T01:12:47.695 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.38882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58478.log 2024-09-15T01:12:47.695 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54786.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54786.log.gz 2024-09-15T01:12:47.697 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65491.log 2024-09-15T01:12:47.698 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58478.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58478.log.gz 2024-09-15T01:12:47.699 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38882.log.gz 2024-09-15T01:12:47.699 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55888.log 2024-09-15T01:12:47.699 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65491.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65491.log.gz 2024-09-15T01:12:47.700 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.48841.log 2024-09-15T01:12:47.700 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.55888.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55888.log.gz 2024-09-15T01:12:47.701 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65200.log 2024-09-15T01:12:47.703 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.1.48841.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68644.log 2024-09-15T01:12:47.704 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65200.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.48841.log.gz 2024-09-15T01:12:47.705 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47944.log 2024-09-15T01:12:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47196.log 2024-09-15T01:12:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47944.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47944.log.gz 2024-09-15T01:12:47.721 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65200.log.gz 2024-09-15T01:12:47.721 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66879.log 2024-09-15T01:12:47.725 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47196.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47196.log.gz 2024-09-15T01:12:47.730 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44676.log 2024-09-15T01:12:47.731 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66879.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66879.log.gz 2024-09-15T01:12:47.734 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53545.log 2024-09-15T01:12:47.735 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44676.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44676.log.gz 2024-09-15T01:12:47.742 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58251.log 2024-09-15T01:12:47.752 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70539.log 2024-09-15T01:12:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58251.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58251.log.gz 2024-09-15T01:12:47.760 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58839.log 2024-09-15T01:12:47.761 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70539.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70539.log.gz 2024-09-15T01:12:47.772 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48130.log.gz 2024-09-15T01:12:47.772 INFO:teuthology.orchestra.run.smithi059.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53545.log.gz 2024-09-15T01:12:47.772 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59742.log 2024-09-15T01:12:47.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50462.log 2024-09-15T01:12:47.772 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58839.log.gz 2024-09-15T01:12:47.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59742.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59742.log.gz 2024-09-15T01:12:47.773 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68969.log 2024-09-15T01:12:47.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50462.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50462.log.gz 2024-09-15T01:12:47.773 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53565.log 2024-09-15T01:12:47.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52133.log 2024-09-15T01:12:47.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53565.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53565.log.gz 2024-09-15T01:12:47.774 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58978.log 2024-09-15T01:12:47.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52133.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52133.log.gz 2024-09-15T01:12:47.789 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72383.log 2024-09-15T01:12:47.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58978.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58978.log.gz 2024-09-15T01:12:47.791 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47648.log 2024-09-15T01:12:47.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72383.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72383.log.gz 2024-09-15T01:12:47.793 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.40030.log 2024-09-15T01:12:47.804 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47648.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70560.log 2024-09-15T01:12:47.819 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.1.40030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48947.log 2024-09-15T01:12:47.822 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70560.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70560.log.gz 2024-09-15T01:12:47.829 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50571.log 2024-09-15T01:12:47.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48947.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48947.log.gz 2024-09-15T01:12:47.830 INFO:teuthology.orchestra.run.smithi059.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.40030.log.gz 2024-09-15T01:12:47.837 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72551.log 2024-09-15T01:12:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56588.log 2024-09-15T01:12:47.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.72551.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72551.log.gz 2024-09-15T01:12:47.851 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50571.log.gz 2024-09-15T01:12:47.866 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49903.log 2024-09-15T01:12:47.867 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56588.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56588.log.gz 2024-09-15T01:12:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44596.log 2024-09-15T01:12:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.49903.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.36782.log.gz 2024-09-15T01:12:47.885 INFO:teuthology.orchestra.run.smithi059.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49903.log.gz 2024-09-15T01:12:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64206.log 2024-09-15T01:12:47.895 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51362.log 2024-09-15T01:12:47.898 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64206.log: 88.2% 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44596.log.gz 2024-09-15T01:12:47.904 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65759.log 2024-09-15T01:12:47.905 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51362.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51362.log.gz 2024-09-15T01:12:47.907 INFO:teuthology.orchestra.run.smithi059.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40884.log.gz 2024-09-15T01:12:47.908 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64206.log.gz 2024-09-15T01:12:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38699.log 2024-09-15T01:12:47.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65759.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.38956.log.gz 2024-09-15T01:12:47.922 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41362.log 2024-09-15T01:12:47.928 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.38699.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65759.log.gz 2024-09-15T01:12:47.938 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53770.log 2024-09-15T01:12:47.940 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41362.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38699.log.gz 2024-09-15T01:12:47.948 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71399.log 2024-09-15T01:12:47.962 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53770.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53770.log.gz 2024-09-15T01:12:47.965 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57833.log 2024-09-15T01:12:47.982 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71399.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41362.log.gz 2024-09-15T01:12:47.984 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46231.log.gz 2024-09-15T01:12:47.997 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37234.log 2024-09-15T01:12:47.997 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47238.log.gz 2024-09-15T01:12:47.998 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57833.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57833.log.gz 2024-09-15T01:12:47.998 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43596.log 2024-09-15T01:12:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.37234.log: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37234.log.gz 2024-09-15T01:12:48.005 INFO:teuthology.orchestra.run.smithi059.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.37626.log.gz 2024-09-15T01:12:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56248.log 2024-09-15T01:12:48.020 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54537.log 2024-09-15T01:12:48.022 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56248.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56248.log.gz 2024-09-15T01:12:48.022 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43596.log.gz 2024-09-15T01:12:48.024 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47318.log 2024-09-15T01:12:48.024 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54537.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54537.log.gz 2024-09-15T01:12:48.024 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61939.log 2024-09-15T01:12:48.035 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47318.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49332.log.gz 2024-09-15T01:12:48.040 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58389.log 2024-09-15T01:12:48.041 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61939.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61939.log.gz 2024-09-15T01:12:48.054 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47924.log 2024-09-15T01:12:48.058 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58389.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58389.log.gz 2024-09-15T01:12:48.058 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50840.log 2024-09-15T01:12:48.063 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47924.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47562.log.gz 2024-09-15T01:12:48.063 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47924.log.gz 2024-09-15T01:12:48.064 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67982.log 2024-09-15T01:12:48.066 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50840.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50840.log.gz 2024-09-15T01:12:48.066 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55489.log 2024-09-15T01:12:48.067 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67982.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67982.log.gz 2024-09-15T01:12:48.068 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35408.log 2024-09-15T01:12:48.068 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67037.log 2024-09-15T01:12:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55489.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55489.log.gz 2024-09-15T01:12:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67239.log 2024-09-15T01:12:48.071 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67037.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67037.log.gz 2024-09-15T01:12:48.071 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62004.log 2024-09-15T01:12:48.071 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35408.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35408.log.gz/var/log/ceph/cluster1-client.mirror.67239.log: 2024-09-15T01:12:48.072 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47482.log 2024-09-15T01:12:48.073 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67239.log.gz 2024-09-15T01:12:48.073 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.61222.log 2024-09-15T01:12:48.081 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47482.log: /var/log/ceph/cluster1-client.mirror.62004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39940.log 2024-09-15T01:12:48.088 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.3.61222.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62004.log.gz 2024-09-15T01:12:48.091 INFO:teuthology.orchestra.run.smithi059.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71016.log 2024-09-15T01:12:48.091 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.39940.log: -- replaced with /var/log/ceph/cluster2-client.mirror.3.61222.log.gz 2024-09-15T01:12:48.092 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63314.log 2024-09-15T01:12:48.094 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71016.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71016.log.gz 2024-09-15T01:12:48.094 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52488.log 2024-09-15T01:12:48.095 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63314.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.39940.log.gz 2024-09-15T01:12:48.097 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63314.log.gz 2024-09-15T01:12:48.097 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63046.log 2024-09-15T01:12:48.099 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52488.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52488.log.gz 2024-09-15T01:12:48.099 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48665.log 2024-09-15T01:12:48.099 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63046.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63046.log.gz 2024-09-15T01:12:48.101 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53499.log 2024-09-15T01:12:48.101 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48665.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48665.log.gz 2024-09-15T01:12:48.102 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40030.log 2024-09-15T01:12:48.102 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53499.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53499.log.gz 2024-09-15T01:12:48.103 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65305.log 2024-09-15T01:12:48.104 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42541.log 2024-09-15T01:12:48.105 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65305.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65305.log.gz 2024-09-15T01:12:48.105 INFO:teuthology.orchestra.run.smithi059.stderr: 74.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40030.log.gz 2024-09-15T01:12:48.106 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47845.log 2024-09-15T01:12:48.107 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.42541.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42541.log.gz 2024-09-15T01:12:48.108 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69533.log 2024-09-15T01:12:48.109 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47845.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47845.log.gz 2024-09-15T01:12:48.110 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69930.log 2024-09-15T01:12:48.110 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69533.log: 85.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69533.log.gz 2024-09-15T01:12:48.111 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48682.log 2024-09-15T01:12:48.112 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69930.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69930.log.gz 2024-09-15T01:12:48.113 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36248.log 2024-09-15T01:12:48.115 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48682.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48682.log.gz 2024-09-15T01:12:48.115 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48298.log 2024-09-15T01:12:48.116 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.36248.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36248.log.gz 2024-09-15T01:12:48.116 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54405.log 2024-09-15T01:12:48.117 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48298.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48298.log.gz 2024-09-15T01:12:48.118 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57560.log 2024-09-15T01:12:48.119 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.54405.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54405.log.gz 2024-09-15T01:12:48.121 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56677.log 2024-09-15T01:12:48.121 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57560.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57560.log.gz 2024-09-15T01:12:48.122 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67824.log 2024-09-15T01:12:48.125 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56677.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56677.log.gz 2024-09-15T01:12:48.125 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48930.log 2024-09-15T01:12:48.125 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67824.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69466.log.gz 2024-09-15T01:12:48.125 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67824.log.gz 2024-09-15T01:12:48.126 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52420.log 2024-09-15T01:12:48.126 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48930.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48930.log.gz 2024-09-15T01:12:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39846.log 2024-09-15T01:12:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50639.log 2024-09-15T01:12:48.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39846.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62818.log 2024-09-15T01:12:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.50639.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50639.log.gz 2024-09-15T01:12:48.131 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52420.log.gz 2024-09-15T01:12:48.136 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49391.log 2024-09-15T01:12:48.136 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.62818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45873.log 2024-09-15T01:12:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49391.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49391.log.gz 2024-09-15T01:12:48.139 INFO:teuthology.orchestra.run.smithi059.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62818.log.gz 2024-09-15T01:12:48.140 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60193.log 2024-09-15T01:12:48.140 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45873.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53682.log 2024-09-15T01:12:48.141 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.60193.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60193.log.gz 2024-09-15T01:12:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63252.log 2024-09-15T01:12:48.150 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53682.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53682.log.gz 2024-09-15T01:12:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70404.log 2024-09-15T01:12:48.157 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63252.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63252.log.gz 2024-09-15T01:12:48.158 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.45208.log 2024-09-15T01:12:48.167 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70404.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70404.log.gz 2024-09-15T01:12:48.167 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67397.log 2024-09-15T01:12:48.179 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.2.45208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69211.log 2024-09-15T01:12:48.179 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67397.log: 86.9% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47648.log.gz 2024-09-15T01:12:48.179 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67397.log.gz 2024-09-15T01:12:48.179 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64619.log 2024-09-15T01:12:48.180 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.69211.log: 85.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69211.log.gz 2024-09-15T01:12:48.181 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46272.log 2024-09-15T01:12:48.190 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64619.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64619.log.gz 2024-09-15T01:12:48.190 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70625.log 2024-09-15T01:12:48.190 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46272.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46272.log.gz 2024-09-15T01:12:48.190 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66812.log 2024-09-15T01:12:48.190 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70625.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70625.log.gz 2024-09-15T01:12:48.190 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52890.log 2024-09-15T01:12:48.192 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66812.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66812.log.gz 2024-09-15T01:12:48.192 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57335.log 2024-09-15T01:12:48.192 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.52890.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52890.log.gz 2024-09-15T01:12:48.193 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46815.log 2024-09-15T01:12:48.194 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57335.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57335.log.gz 2024-09-15T01:12:48.196 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70746.log 2024-09-15T01:12:48.263 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46815.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46815.log.gz 2024-09-15T01:12:48.263 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35572.log 2024-09-15T01:12:48.268 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39866.log 2024-09-15T01:12:48.268 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35572.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35572.log.gz 2024-09-15T01:12:48.268 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41158.log 2024-09-15T01:12:48.268 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.39866.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47606.log 2024-09-15T01:12:48.268 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41158.log: gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-15T01:12:48.269 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47606.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41158.log.gz 2024-09-15T01:12:48.269 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47606.log.gz 2024-09-15T01:12:48.269 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68739.log 2024-09-15T01:12:48.269 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2.log: 87.5% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-15T01:12:48.269 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57356.log 2024-09-15T01:12:48.269 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68739.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68739.log.gz 2024-09-15T01:12:48.269 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36332.log 2024-09-15T01:12:48.270 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57356.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57356.log.gz 2024-09-15T01:12:48.275 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47019.log 2024-09-15T01:12:48.275 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.admin.36332.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36332.log.gz 2024-09-15T01:12:48.277 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.42642.log 2024-09-15T01:12:48.277 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47019.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47019.log.gz 2024-09-15T01:12:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.42177.log 2024-09-15T01:12:48.308 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.1.42642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35573.log 2024-09-15T01:12:48.314 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.1.42177.log: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.42177.log.gz 2024-09-15T01:12:48.320 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68644.log.gz 2024-09-15T01:12:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66902.log 2024-09-15T01:12:48.326 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35573.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35573.log.gz 2024-09-15T01:12:48.339 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65284.log 2024-09-15T01:12:48.340 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66902.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66902.log.gz 2024-09-15T01:12:48.353 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54111.log 2024-09-15T01:12:48.354 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65284.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65284.log.gz 2024-09-15T01:12:48.367 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58937.log 2024-09-15T01:12:48.367 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54111.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54111.log.gz 2024-09-15T01:12:48.392 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63792.log 2024-09-15T01:12:48.393 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58937.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58937.log.gz 2024-09-15T01:12:48.397 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39846.log.gz 2024-09-15T01:12:48.403 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40233.log 2024-09-15T01:12:48.418 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63792.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63792.log.gz 2024-09-15T01:12:48.421 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68827.log 2024-09-15T01:12:48.426 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40233.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40233.log.gz 2024-09-15T01:12:48.431 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43639.log 2024-09-15T01:12:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68827.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68827.log.gz 2024-09-15T01:12:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71074.log 2024-09-15T01:12:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43639.log: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59408.log 2024-09-15T01:12:48.459 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.71074.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43639.log.gz 2024-09-15T01:12:48.466 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71074.log.gz 2024-09-15T01:12:48.472 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66654.log 2024-09-15T01:12:48.473 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59408.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39866.log.gz 2024-09-15T01:12:48.474 INFO:teuthology.orchestra.run.smithi059.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59408.log.gz 2024-09-15T01:12:48.481 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52464.log 2024-09-15T01:12:48.495 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66654.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66654.log.gz 2024-09-15T01:12:48.496 INFO:teuthology.orchestra.run.smithi059.stderr: 94.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56612.log 2024-09-15T01:12:48.496 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52464.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52464.log.gz 2024-09-15T01:12:48.496 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.42642.log.gz 2024-09-15T01:12:48.498 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53637.log 2024-09-15T01:12:48.499 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56612.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56612.log.gz 2024-09-15T01:12:48.513 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67082.log 2024-09-15T01:12:48.513 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53637.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53637.log.gz 2024-09-15T01:12:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61809.log 2024-09-15T01:12:48.519 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67082.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67082.log.gz 2024-09-15T01:12:48.519 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44198.log 2024-09-15T01:12:48.519 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.61809.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61809.log.gz 2024-09-15T01:12:48.520 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71192.log 2024-09-15T01:12:48.525 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44198.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44198.log.gz 2024-09-15T01:12:48.526 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71399.log.gz 2024-09-15T01:12:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71904.log 2024-09-15T01:12:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56699.log 2024-09-15T01:12:48.527 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71904.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71904.log.gz 2024-09-15T01:12:48.527 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58795.log 2024-09-15T01:12:48.528 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56699.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56699.log.gz 2024-09-15T01:12:48.528 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71192.log.gz 2024-09-15T01:12:48.529 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48261.log 2024-09-15T01:12:48.529 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66789.log 2024-09-15T01:12:48.530 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48261.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48261.log.gz 2024-09-15T01:12:48.530 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58795.log.gz 2024-09-15T01:12:48.530 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44796.log 2024-09-15T01:12:48.531 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48071.log 2024-09-15T01:12:48.531 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66789.log.gz 2024-09-15T01:12:48.532 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.44796.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44796.log.gz 2024-09-15T01:12:48.532 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71036.log 2024-09-15T01:12:48.534 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48071.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48071.log.gz 2024-09-15T01:12:48.534 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65367.log 2024-09-15T01:12:48.534 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67509.log 2024-09-15T01:12:48.535 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.65367.log: /var/log/ceph/cluster2-client.mirror.71036.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65367.log.gz 2024-09-15T01:12:48.535 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55981.log 2024-09-15T01:12:48.537 INFO:teuthology.orchestra.run.smithi059.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71036.log.gz 2024-09-15T01:12:48.537 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67127.log 2024-09-15T01:12:48.540 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.55981.log: /var/log/ceph/cluster1-client.mirror.67509.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55981.log.gz 2024-09-15T01:12:48.540 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47503.log 2024-09-15T01:12:48.541 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.67127.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67127.log.gz 2024-09-15T01:12:48.541 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38904.log 2024-09-15T01:12:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47503.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47503.log.gz 2024-09-15T01:12:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63025.log 2024-09-15T01:12:48.543 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67509.log.gz 2024-09-15T01:12:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.38904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41206.log 2024-09-15T01:12:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56023.log 2024-09-15T01:12:48.545 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.41206.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63025.log.gz 2024-09-15T01:12:48.545 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35952.log 2024-09-15T01:12:48.545 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/cluster2-client.mirror.56023.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36662.log 2024-09-15T01:12:48.545 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.35952.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35952.log.gz 2024-09-15T01:12:48.545 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56023.log.gz 2024-09-15T01:12:48.553 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47482.log.gz 2024-09-15T01:12:48.553 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41206.log.gz 2024-09-15T01:12:48.553 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.36662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46735.log 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72004.log 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48357.log 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68969.log.gz/var/log/ceph/cluster2-client.mirror.72004.log: 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46735.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72004.log.gz 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46735.log.gz 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/cluster2-client.mirror.61653.log 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48357.log: /var/log/ceph/cluster2-client.mirror.61653.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48357.log.gz 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70057.log 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr: 73.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36662.log.gz 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65739.log 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr: 92.4%/var/log/ceph/cluster2-client.mirror.70057.log: -- replaced with /var/log/ceph/cluster2-client.mirror.61653.log.gz 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70057.log.gz 2024-09-15T01:12:48.554 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57896.log 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.65739.log: -- /var/log/ceph/cluster2-client.mirror.65014.log 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.57896.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65739.log.gz 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70785.log 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.65014.log: /var/log/ceph/cluster2-client.mirror.48051.log 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.70785.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57896.log.gz 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr: 83.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66744.log 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70785.log.gz 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/cluster2-client.mirror.48051.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48598.log 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.66744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47440.log 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66744.log.gz 2024-09-15T01:12:48.555 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.48598.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65014.log.gz 2024-09-15T01:12:48.566 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48598.log.gz 2024-09-15T01:12:48.566 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59901.log 2024-09-15T01:12:48.566 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.47440.log: -- /var/log/ceph/cluster2-client.mirror.64144.log 2024-09-15T01:12:48.566 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.59901.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47440.log.gz 2024-09-15T01:12:48.566 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64973.log 2024-09-15T01:12:48.566 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64144.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59901.log.gz 2024-09-15T01:12:48.566 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70909.log 2024-09-15T01:12:48.566 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64144.log.gz 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42468.log 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64973.log: /var/log/ceph/cluster1-client.mirror.70909.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70909.log.gz 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64973.log.gz 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40787.log 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42468.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64911.log 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40787.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42468.log.gz 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56765.log 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.64911.log: -- /var/log/ceph/cluster1-client.mirror.53862.log 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.56765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51434.log 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64911.log.gz 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56765.log.gz 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.53862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50774.log 2024-09-15T01:12:48.567 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53862.log.gz 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.51434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63480.log 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50774.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51997.log 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51434.log.gz 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53973.log 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.63480.log: /var/log/ceph/cluster1-client.mirror.51997.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50774.log.gz 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63480.log.gz 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40787.log.gz 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51997.log.gz 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49610.log 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54742.log 2024-09-15T01:12:48.568 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.53973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47786.log 2024-09-15T01:12:48.595 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49610.log: /var/log/ceph/cluster2-client.mirror.54742.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53973.log.gz 2024-09-15T01:12:48.595 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49610.log.gz 2024-09-15T01:12:48.595 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54159.log 2024-09-15T01:12:48.595 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47786.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54742.log.gz 2024-09-15T01:12:48.595 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47786.log.gz 2024-09-15T01:12:48.595 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54312.log 2024-09-15T01:12:48.595 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49152.log 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54159.log.gz 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.54312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.70785.log 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54312.log.gzgzip 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69811.log 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49152.log: /var/log/ceph/cluster2-client.mirror.0.70785.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49152.log.gz 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60304.log 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.69811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32449.log 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60304.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69811.log.gz 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60304.log.gz 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55332.log 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.admin.32449.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32449.log.gz 2024-09-15T01:12:48.596 INFO:teuthology.orchestra.run.smithi059.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.70785.log.gz 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58566.log 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49512.log 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.58566.log: /var/log/ceph/cluster1-client.mirror.55332.log: 87.3% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55332.log.gz 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58566.log.gz 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57584.log 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56861.log 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.57584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53412.log 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57584.log.gz 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56861.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70363.log 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56861.log.gz/var/log/ceph/cluster1-client.mirror.53412.log: 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53412.log.gz 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39162.log 2024-09-15T01:12:48.597 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64123.log 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70363.log.gz 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39162.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49512.log.gz 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62674.log 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70462.log 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44089.log 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.70462.log: /var/log/ceph/cluster2-client.mirror.62674.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64123.log.gz 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62674.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56386.log 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.44089.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68213.log 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.56386.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39162.log.gz 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56386.log.gz 2024-09-15T01:12:48.598 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60482.log 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.68213.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68213.log.gz 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70462.log.gz 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64932.log 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.64932.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64932.log.gz 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60482.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60482.log.gz 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40366.log 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53027.log 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.40366.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44089.log.gz 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39409.log 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.53027.log: -- /var/log/ceph/cluster2-client.mirror.49354.log 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.39409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62437.log 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.49354.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53027.log.gz 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49354.log.gz 2024-09-15T01:12:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41406.log 2024-09-15T01:12:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.62437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43184.log 2024-09-15T01:12:48.600 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62437.log.gz 2024-09-15T01:12:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.41406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50732.log 2024-09-15T01:12:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43184.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46757.log 2024-09-15T01:12:48.600 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43184.log.gz 2024-09-15T01:12:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58727.log 2024-09-15T01:12:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60755.log 2024-09-15T01:12:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.58727.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58727.log.gz 2024-09-15T01:12:48.600 INFO:teuthology.orchestra.run.smithi059.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50732.log.gz 2024-09-15T01:12:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71458.log 2024-09-15T01:12:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.60755.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60755.log.gz 2024-09-15T01:12:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.36702.log 2024-09-15T01:12:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.71458.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71458.log.gz 2024-09-15T01:12:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42915.log 2024-09-15T01:12:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.3.36702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68117.log 2024-09-15T01:12:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.42915.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35168.log 2024-09-15T01:12:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.68117.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68117.log.gz 2024-09-15T01:12:48.607 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42915.log.gz 2024-09-15T01:12:48.610 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52311.log 2024-09-15T01:12:48.610 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36782.log 2024-09-15T01:12:48.610 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.52311.log: /var/log/ceph/cluster2-client.admin.35168.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35168.log.gz 2024-09-15T01:12:48.610 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52311.log.gz 2024-09-15T01:12:48.610 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38771.log 2024-09-15T01:12:48.610 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46778.log 2024-09-15T01:12:48.612 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.36782.log: 82.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36782.log.gz 2024-09-15T01:12:48.612 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.38771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43273.log 2024-09-15T01:12:48.614 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50820.log 2024-09-15T01:12:48.614 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.43273.log: /var/log/ceph/cluster2-client.mirror.46778.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46778.log.gz 2024-09-15T01:12:48.614 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62389.log 2024-09-15T01:12:48.615 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.50820.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50820.log.gz 2024-09-15T01:12:48.615 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38771.log.gz 2024-09-15T01:12:48.617 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45609.log 2024-09-15T01:12:48.617 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster1-client.mirror.62389.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62389.log.gz 2024-09-15T01:12:48.617 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47992.log 2024-09-15T01:12:48.617 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.45609.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45609.log.gz 2024-09-15T01:12:48.621 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43273.log.gz 2024-09-15T01:12:48.621 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40745.log 2024-09-15T01:12:48.621 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.47992.log: 80.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57874.log 2024-09-15T01:12:48.621 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47992.log.gz 2024-09-15T01:12:48.622 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46314.log 2024-09-15T01:12:48.622 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40745.log: /var/log/ceph/cluster1-client.mirror.57874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40809.log 2024-09-15T01:12:48.626 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.46314.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57874.log.gz 2024-09-15T01:12:48.626 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57942.log 2024-09-15T01:12:48.636 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cluster2-client.mirror.40809.log: /var/log/ceph/cluster2-client.mirror.57942.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57942.log.gz 2024-09-15T01:12:48.693 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45873.log.gz 2024-09-15T01:12:48.718 INFO:teuthology.orchestra.run.smithi059.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.45208.log.gz 2024-09-15T01:12:48.756 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47318.log.gz 2024-09-15T01:12:48.768 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40745.log.gz 2024-09-15T01:12:48.797 INFO:teuthology.orchestra.run.smithi059.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70746.log.gz 2024-09-15T01:12:48.844 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.36702.log.gz 2024-09-15T01:12:48.856 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48051.log.gz 2024-09-15T01:12:49.039 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46757.log.gz 2024-09-15T01:12:49.042 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39409.log.gz 2024-09-15T01:12:49.042 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40366.log.gz 2024-09-15T01:12:49.111 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46314.log.gz 2024-09-15T01:12:50.968 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40410.log.gz 2024-09-15T01:12:52.594 INFO:teuthology.orchestra.run.smithi002.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-15T01:12:52.690 INFO:teuthology.orchestra.run.smithi002.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-15T01:12:53.197 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41406.log.gz 2024-09-15T01:12:53.263 INFO:teuthology.orchestra.run.smithi059.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-15T01:12:53.695 INFO:teuthology.orchestra.run.smithi002.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-15T01:12:53.696 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-15T01:12:53.697 INFO:teuthology.orchestra.run.smithi002.stderr:real 0m9.819s 2024-09-15T01:12:53.697 INFO:teuthology.orchestra.run.smithi002.stderr:user 0m28.010s 2024-09-15T01:12:53.697 INFO:teuthology.orchestra.run.smithi002.stderr:sys 0m0.971s 2024-09-15T01:12:54.003 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39910.log.gz 2024-09-15T01:12:56.005 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37755.log.gz 2024-09-15T01:12:56.418 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41848.log.gz 2024-09-15T01:12:57.963 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40809.log.gz 2024-09-15T01:12:58.514 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39516.log.gz 2024-09-15T01:13:05.378 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38904.log.gz 2024-09-15T01:14:53.961 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-15T01:15:02.101 INFO:teuthology.orchestra.run.smithi059.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-15T01:15:23.494 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-15T01:15:23.494 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-15T01:15:23.495 INFO:teuthology.orchestra.run.smithi059.stderr:real 2m39.575s 2024-09-15T01:15:23.495 INFO:teuthology.orchestra.run.smithi059.stderr:user 4m19.835s 2024-09-15T01:15:23.495 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m10.400s 2024-09-15T01:15:23.495 INFO:tasks.ceph:Archiving logs... 2024-09-15T01:15:23.495 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845/remote/smithi002/log 2024-09-15T01:15:23.497 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T01:15:27.234 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845/remote/smithi059/log 2024-09-15T01:15:27.236 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T01:16:04.242 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T01:16:04.254 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T01:16:04.254 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T01:16:04.257 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T01:16:04.357 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T01:16:04.368 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T01:16:04.369 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T01:16:04.372 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T01:16:04.387 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-09-15T01:16:04.390 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-09-15T01:16:04.391 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T01:16:04.392 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-09-15T01:16:04.392 INFO:teuthology.orchestra.run.smithi002.stdout:^* hv01.front.sepia.ceph.com 3 6 377 6 -44us[ -36us] +/- 46ms 2024-09-15T01:16:04.392 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 137 +302us[ +319us] +/- 60ms 2024-09-15T01:16:04.392 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -383us[ -375us] +/- 68ms 2024-09-15T01:16:04.392 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T01:16:04.447 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T01:16:04.448 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-09-15T01:16:04.448 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv01.front.sepia.ceph.com 3 6 377 6 +3650ns[-1648ns] +/- 46ms 2024-09-15T01:16:04.448 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 +294us[ +289us] +/- 62ms 2024-09-15T01:16:04.448 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -346us[ -351us] +/- 68ms 2024-09-15T01:16:04.448 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T01:16:04.448 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T01:16:04.460 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T01:16:04.461 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T01:16:04.485 DEBUG:teuthology.orchestra.run.smithi002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T01:16:04.555 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T01:16:04.660 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T01:16:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:16:04.690 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T01:16:04.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:16:04.721 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T01:16:04.732 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T01:16:04.760 INFO:teuthology.task.internal:Duration was 2898.725329 seconds 2024-09-15T01:16:04.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T01:16:04.794 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T01:16:04.794 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T01:16:04.796 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T01:16:04.836 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T01:16:04.861 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T01:16:04.973 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T01:16:04.973 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2024-09-15T01:16:04.973 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T01:16:05.035 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2024-09-15T01:16:05.035 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T01:16:05.058 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T01:16:05.058 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T01:16:05.078 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T01:16:05.128 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T01:16:05.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T01:16:05.146 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T01:16:05.288 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T01:16:05.300 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T01:16:05.300 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T01:16:05.326 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T01:16:05.365 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T01:16:05.376 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T01:16:05.379 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T01:16:05.400 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2024-09-15T01:16:05.445 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-09-15T01:16:05.459 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T01:16:05.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:16:05.475 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T01:16:05.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:16:05.515 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T01:16:05.525 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T01:16:05.526 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845/remote/smithi002 2024-09-15T01:16:05.526 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T01:16:05.561 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904845/remote/smithi059 2024-09-15T01:16:05.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T01:16:05.596 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T01:16:05.596 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T01:16:05.598 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T01:16:05.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T01:16:05.659 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T01:16:05.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T01:16:05.679 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T01:16:05.679 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T01:16:05.681 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T01:16:05.694 INFO:teuthology.orchestra.run.smithi002.stdout: 265978 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 01:16 /home/ubuntu/cephtest 2024-09-15T01:16:05.702 INFO:teuthology.orchestra.run.smithi059.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 01:16 /home/ubuntu/cephtest 2024-09-15T01:16:05.703 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T01:16:05.716 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T01:16:05.800 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 2898.725329399109 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-15T01:16:05.800 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T01:16:05.873 INFO:teuthology.run:pass